airflow.providers.google.cloud.example_dags.example_bigtable¶
Example Airflow DAG that creates and performs following operations on Cloud Bigtable: - creates an Instance - creates a Table - updates Cluster - waits for Table replication completeness - deletes the Table - deletes the Instance
This DAG relies on the following environment variables:
GCP_PROJECT_ID - Google Cloud project
CBT_INSTANCE_ID - desired ID of a Cloud Bigtable instance
CBT_INSTANCE_DISPLAY_NAME - desired human-readable display name of the Instance
- CBT_INSTANCE_TYPE - type of the Instance, e.g. 1 for DEVELOPMENT
See https://googleapis.github.io/google-cloud-python/latest/bigtable/instance.html#google.cloud.bigtable.instance.Instance # noqa E501
CBT_INSTANCE_LABELS - labels to add for the Instance
CBT_CLUSTER_ID - desired ID of the main Cluster created for the Instance
- CBT_CLUSTER_ZONE - zone in which main Cluster will be created. e.g. europe-west1-b
See available zones: https://cloud.google.com/bigtable/docs/locations
CBT_CLUSTER_NODES - initial amount of nodes of the Cluster
CBT_CLUSTER_NODES_UPDATED - amount of nodes for BigtableClusterUpdateOperator
- CBT_CLUSTER_STORAGE_TYPE - storage for the Cluster, e.g. 1 for SSD
See https://googleapis.github.io/google-cloud-python/latest/bigtable/instance.html#google.cloud.bigtable.instance.Instance.cluster # noqa E501
CBT_TABLE_ID - desired ID of the Table
CBT_POKE_INTERVAL - number of seconds between every attempt of Sensor check