• Home
• tests.system.google
• tests.system.google.cloud
• tests.system.google.cloud.bigquery
• tests.system.google.cloud.bigquery.example_bigquery_to_postgres

tests.system.google.cloud.bigquery.example_bigquery_to_postgres

Example Airflow DAG for Google BigQuery service.

This DAG relies on the following OS environment variables

• AIRFLOW__API__GOOGLE_KEY_PATH - Path to service account key file. Note, you can skip this variable if you run this DAG in a Composer environment.

Attributes

ENV_ID
PROJECT_ID
DAG_ID
REGION
ZONE
NETWORK
CONNECTION_ID
CONNECTION_TYPE
BIGQUERY_DATASET_NAME
BIGQUERY_TABLE
SOURCE_OBJECT_NAME
BATCH_SIZE
UPLOAD_DATA_TO_BIGQUERY
DB_NAME
DB_PORT
DB_USER_NAME
DB_USER_PASSWORD
SCHEMA
SETUP_POSTGRES_COMMAND
SQL_TABLE
SQL_CREATE_TABLE
GCE_MACHINE_TYPE
GCE_INSTANCE_NAME
GCE_INSTANCE_BODY
FIREWALL_RULE_NAME
CREATE_FIREWALL_RULE_COMMAND
DELETE_FIREWALL_RULE_COMMAND
log
create_bigquery_dataset
test_run

Module Contents

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.ENV_ID

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.PROJECT_ID

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.DAG_ID = 'bigquery_to_postgres'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.REGION = 'us-west2'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.ZONE = 'us-west2-a'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.NETWORK = 'default'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.CONNECTION_ID = ''

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.CONNECTION_TYPE = 'postgres'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.BIGQUERY_DATASET_NAME = ''

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.BIGQUERY_TABLE = 'test_table'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.SOURCE_OBJECT_NAME = 'gs://airflow-system-tests-resources/bigquery/salaries_1k.csv'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.BATCH_SIZE = 500

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.UPLOAD_DATA_TO_BIGQUERY

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.DB_NAME = 'testdb'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.DB_PORT = 5432

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.DB_USER_NAME = 'root'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.DB_USER_PASSWORD = 'demo_password'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.SCHEMA

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.SETUP_POSTGRES_COMMAND = Multiline-String

Show Value

"""
sudo apt update &&
sudo apt install -y docker.io &&
sudo docker run -d -p 5432:5432 --name testdb     -e PGUSER=root     -e POSTGRES_USER=root     -e POSTGRES_PASSWORD=demo_password     -e POSTGRES_DB=testdb     postgres
"""

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.SQL_TABLE = 'test_table'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.SQL_CREATE_TABLE = 'CREATE TABLE IF NOT EXISTS test_table (emp_name VARCHAR(64), salary FLOAT)'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.GCE_MACHINE_TYPE = 'n1-standard-1'

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.GCE_INSTANCE_NAME = ''

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.GCE_INSTANCE_BODY

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.FIREWALL_RULE_NAME = ''

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.CREATE_FIREWALL_RULE_COMMAND

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.DELETE_FIREWALL_RULE_COMMAND

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.log

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.create_bigquery_dataset

tests.system.google.cloud.bigquery.example_bigquery_to_postgres.test_run

Previous Next

Was this entry helpful?