Google Dataprep Operators

Dataprep is the intelligent cloud data service to visually explore, clean, and prepare data for analysis and machine learning. Service can be use to explore and transform raw data from disparate and/or large datasets into clean and structured data for further analysis and processing. Dataprep Job is an internal object encoding the information necessary to run a part of a Cloud Dataprep job group. For more information about the service visit Google Dataprep API documentation

Before you begin

Before using Dataprep within Airflow you need to authenticate your account with TOKEN. To get connection Dataprep with Airflow you need Dataprep token. Please follow Dataprep instructions to do it.

TOKEN should be added to the Connection in Airflow in JSON format. You can check Managing Connections

The DataprepRunJobGroupOperator will run specified job. Operator required a recipe id. To identify the recipe id please use API documentation for runJobGroup E.g. if the URL is /flows/10?recipe=7, the recipe id is 7. The recipe cannot be created via this operator. It can be created only via UI which is available here. Some of parameters can be override by DAG's body request. How to do it is shown in example dag.

See following example: Set values for these fields: .. code-block:

Connection Id: "your_conn_id"
Extra: {"extra__dataprep__token": "TOKEN",
        "extra__dataprep__base_url": "https://api.clouddataprep.com"}

Prerequisite Tasks

Run Job Group

Operator task is to create a job group, which launches the specified job as the authenticated user. This performs the same action as clicking on the Run Job button in the application.

To get information about jobs within a Cloud Dataprep job use: DataprepRunJobGroupOperator

Example usage:

airflow/providers/google/cloud/example_dags/example_dataprep.pyView Source

run_job_group = DataprepRunJobGroupOperator(task_id="run_job_group", body_request=DATA)

Get Jobs For Job Group

Operator task is to get information about the batch jobs within a Cloud Dataprep job.

To get information about jobs within a Cloud Dataprep job use: DataprepGetJobsForJobGroupOperator

Example usage:

airflow/providers/google/cloud/example_dags/example_dataprep.pyView Source

get_jobs_for_job_group = DataprepGetJobsForJobGroupOperator(
    task_id="get_jobs_for_job_group", job_id=DATAPREP_JOB_ID
)

Get Job Group

Operator task is to get the specified job group. A job group is a job that is executed from a specific node in a flow.

To get information about jobs within a Cloud Dataprep job use: DataprepGetJobGroupOperator

Example usage:

airflow/providers/google/cloud/example_dags/example_dataprep.pyView Source

get_job_group = DataprepGetJobGroupOperator(
    task_id="get_job_group",
    job_group_id=DATAPREP_JOB_ID,
    embed="",
    include_deleted=False,
)

Was this entry helpful?