tests.system.providers.google.cloud.gcs.example_gcs_to_gcs

Example Airflow DAG for Google Cloud Storage GCSSynchronizeBucketsOperator and GCSToGCSOperator operators.

Module Contents

Functions

create_workdir()

Task creates working directory. The logic behind this task is a workaround that provides sustainable

Attributes

ENV_ID

PROJECT_ID

DAG_ID

BUCKET_NAME_SRC

BUCKET_NAME_DST

HOME

PREFIX

test_run

tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.ENV_ID[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.PROJECT_ID[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.DAG_ID = 'gcs_to_gcs'[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.BUCKET_NAME_SRC[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.BUCKET_NAME_DST[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.HOME = '/home/airflow/gcs'[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.PREFIX[source]
tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.create_workdir()[source]

Task creates working directory. The logic behind this task is a workaround that provides sustainable execution in Composer environment: local files can be safely shared among tasks if they are located within ‘/home/airflow/gcs/data/’ folder which is mounted to GCS bucket under the hood (https://cloud.google.com/composer/docs/composer-2/cloud-storage). Otherwise, worker nodes don’t share local path and thus files created by one task aren’t guaranteed to be accessible be others.

tests.system.providers.google.cloud.gcs.example_gcs_to_gcs.test_run[source]

Was this entry helpful?