tests.system.google.cloud.dataproc.example_dataproc_cluster_generator¶

Example Airflow DAG testing Managed Spark operators for managing a cluster and submitting jobs.

Attributes¶

`ENV_ID`
`DAG_ID`
`PROJECT_ID`
`BUCKET_NAME`
`RESOURCE_DATA_BUCKET`
`INIT_FILE`
`GCS_INIT_FILE`
`CLUSTER_NAME_BASE`
`CLUSTER_NAME_FULL`
`CLUSTER_NAME`
`REGION`
`ZONE`
`CLUSTER_GENERATOR_CONFIG`
`create_bucket`
`test_run`

Module Contents¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.ENV_ID[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.DAG_ID = 'dataproc_cluster_generation'[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.PROJECT_ID[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.BUCKET_NAME[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.RESOURCE_DATA_BUCKET = 'airflow-system-tests-resources'[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.INIT_FILE = 'pip-install.sh'[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.GCS_INIT_FILE = 'gs://airflow-system-tests-resources/dataproc/pip-install.sh'[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.CLUSTER_NAME_BASE = ''[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.CLUSTER_NAME_FULL[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.CLUSTER_NAME = ''[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.REGION = 'us-east4'[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.ZONE = 'us-east4-a'[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.CLUSTER_GENERATOR_CONFIG[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.create_bucket[source]¶

tests.system.google.cloud.dataproc.example_dataproc_cluster_generator.test_run[source]¶

Was this entry helpful?