tests.system.google.cloud.dataflow.example_dataflow_java_streaming

Example Airflow DAG for testing Google Dataflow Beam Pipeline Operator with Java(streaming).

Important Note:

This test downloads Java JAR file from the public bucket. In case the JAR file cannot be downloaded or is not compatible with the Java version used in the test. There is no streaming pipeline example for Apache Beam Java SDK, the source code and build instructions are located in providers/google/tests/system/google/cloud/dataflow/resources/java_streaming_src/.

You can follow the instructions on how to pack a self-executing jar here: https://beam.apache.org/documentation/runners/dataflow/

Requirements:

These operators require the gcloud command and Java’s JRE to run.

Attributes

ENV_ID

PROJECT_ID

DAG_ID

LOCATION

BUCKET_NAME

GCS_TMP

GCS_OUTPUT

RESOURCE_BUCKET

JAR_FILE_NAME

GCS_JAR_PATH

IS_COMPOSER

LOCAL_JAR

REMOTE_JAR_FILE_PATH

OUTPUT_TOPIC_ID

OUTPUT_TOPIC_ID_2

INPUT_TOPIC

OUTPUT_TOPIC

OUTPUT_TOPIC_2

create_bucket

test_run

Module Contents

tests.system.google.cloud.dataflow.example_dataflow_java_streaming.ENV_ID[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.PROJECT_ID[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.DAG_ID = 'dataflow_java_streaming'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.LOCATION = 'europe-west3'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.BUCKET_NAME = 'bucket-dataflow_java_streaming-Uninferable'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.GCS_TMP = 'gs://bucket-dataflow_java_streaming-Uninferable/temp'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.GCS_OUTPUT = 'gs://bucket-dataflow_java_streaming-Uninferable/DF_OUT'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.RESOURCE_BUCKET = 'airflow-system-tests-resources'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.JAR_FILE_NAME = 'stream-pubsub-example-bundled-v-0.1.jar'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.GCS_JAR_PATH = 'gs://airflow-system-tests-resources/dataflow/java/stream-pubsub-example-bundled-v-0.1.jar'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.IS_COMPOSER[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.LOCAL_JAR = 'gcs/data/stream-pubsub-example-bundled-v-0.1.jar'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.REMOTE_JAR_FILE_PATH = 'dataflow/java/stream-pubsub-example-bundled-v-0.1.jar'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.OUTPUT_TOPIC_ID = 'tp-Uninferable-out'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.OUTPUT_TOPIC_ID_2 = 'tp-2-Uninferable-out'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.INPUT_TOPIC = 'projects/pubsub-public-data/topics/taxirides-realtime'[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.OUTPUT_TOPIC[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.OUTPUT_TOPIC_2[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.create_bucket[source]
tests.system.google.cloud.dataflow.example_dataflow_java_streaming.test_run[source]

Was this entry helpful?