tests.system.providers.amazon.aws.example_glue

Module Contents

Functions

delete_logs(job_id, glue_crawler_name)

Glue generates four Cloudwatch log groups and multiple log streams and leaves them.

glue_cleanup(glue_crawler_name, glue_job_name, ...)

set_up(env_id, role_arn)

Attributes

DAG_ID

ROLE_ARN_KEY

sys_test_context_task

EXAMPLE_CSV

EXAMPLE_SCRIPT

test_context

test_run

tests.system.providers.amazon.aws.example_glue.DAG_ID = example_glue[source]
tests.system.providers.amazon.aws.example_glue.ROLE_ARN_KEY = ROLE_ARN[source]
tests.system.providers.amazon.aws.example_glue.sys_test_context_task[source]
tests.system.providers.amazon.aws.example_glue.EXAMPLE_CSV = Multiline-String[source]
Show Value
1apple,0.5
2milk,2.5
3bread,4.0
tests.system.providers.amazon.aws.example_glue.EXAMPLE_SCRIPT = Multiline-String[source]
Show Value
1from pyspark.context import SparkContext
2from awsglue.context import GlueContext
3
4glueContext = GlueContext(SparkContext.getOrCreate())
5datasource = glueContext.create_dynamic_frame.from_catalog(
6             database='{db_name}', table_name='input')
7print('There are %s items in the table' % datasource.count())
8
9datasource.toDF().write.format('csv').mode("append").save('s3://{bucket_name}/output')
tests.system.providers.amazon.aws.example_glue.delete_logs(job_id, glue_crawler_name)[source]

Glue generates four Cloudwatch log groups and multiple log streams and leaves them.

tests.system.providers.amazon.aws.example_glue.glue_cleanup(glue_crawler_name, glue_job_name, glue_db_name)[source]
tests.system.providers.amazon.aws.example_glue.set_up(env_id, role_arn)[source]
tests.system.providers.amazon.aws.example_glue.test_context[source]
tests.system.providers.amazon.aws.example_glue.test_run[source]

Was this entry helpful?