tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming¶

Example Airflow DAG for testing Google Dataflow to create pipelines.

Attributes¶

`DAG_ID`
`ENV_ID`
`GCP_PROJECT_ID`
`GCP_LOCATION`
`PIPELINE_NAME`
`PIPELINE_JOB_NAME`
`PIPELINE_TYPE`
`RESOURCE_DATA_BUCKET`
`BUCKET_NAME`
`INPUT_TOPIC_ID`
`INPUT_TOPIC_PROJECT_ID`
`INPUT_TOPIC`
`SUBSCRIPTION_NAME`
`OUTPUT_TOPIC_ID`
`OUTPUT_TOPIC`
`create_output_pub_sub_topic`
`test_run`

Module Contents¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.DAG_ID = 'dataflow_pipeline_streaming'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.ENV_ID[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.GCP_PROJECT_ID[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.GCP_LOCATION = 'europe-central2'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.PIPELINE_NAME[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.PIPELINE_JOB_NAME = ''[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.PIPELINE_TYPE = 'PIPELINE_TYPE_STREAMING'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.RESOURCE_DATA_BUCKET = 'airflow-system-tests-resources'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.BUCKET_NAME[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.INPUT_TOPIC_ID = 'taxirides-realtime'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.INPUT_TOPIC_PROJECT_ID = 'pubsub-public-data'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.INPUT_TOPIC = 'projects/pubsub-public-data/topics/taxirides-realtime'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.SUBSCRIPTION_NAME = 'taxirides-sub'[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.OUTPUT_TOPIC_ID[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.OUTPUT_TOPIC[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.create_output_pub_sub_topic[source]¶

tests.system.google.cloud.dataflow.example_dataflow_pipeline_streaming.test_run[source]¶

Was this entry helpful?