tests.system.google.cloud.gcs.example_gcs_to_gcs¶

Example Airflow DAG for Google Cloud Storage GCSSynchronizeBucketsOperator and GCSToGCSOperator operators.

Attributes¶

`ENV_ID`
`PROJECT_ID`
`DAG_ID`
`BUCKET_NAME_SRC`
`BUCKET_NAME_DST`
`HOME`
`PREFIX`
`test_run`

Functions¶

create_workdir()

Task creates working directory. The logic behind this task is a workaround that provides sustainable

Module Contents¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.ENV_ID[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.PROJECT_ID[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.DAG_ID = 'gcs_to_gcs'[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.BUCKET_NAME_SRC = 'bucket_gcs_to_gcs_Uninferable'[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.BUCKET_NAME_DST = 'bucket_dst_gcs_to_gcs_Uninferable'[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.HOME = '/home/airflow/gcs'[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.PREFIX = '/home/airflow/gcs/data/gcs_to_gcs_Uninferable/'[source]¶

tests.system.google.cloud.gcs.example_gcs_to_gcs.create_workdir()[source]¶

Task creates working directory. The logic behind this task is a workaround that provides sustainable execution in Composer environment: local files can be safely shared among tasks if they are located within ‘/home/airflow/gcs/data/’ folder which is mounted to GCS bucket under the hood (https://cloud.google.com/composer/docs/composer-2/cloud-storage). Otherwise, worker nodes don’t share local path and thus files created by one task aren’t guaranteed to be accessible be others.

tests.system.google.cloud.gcs.example_gcs_to_gcs.test_run[source]¶