Google Dataprep Operators¶
Dataprep is the intelligent cloud data service to visually explore, clean, and prepare data for analysis and machine learning. Service can be use to explore and transform raw data from disparate and/or large datasets into clean and structured data for further analysis and processing. Dataprep Job is an internal object encoding the information necessary to run a part of a Cloud Dataprep job group. For more information about the service visit Google Dataprep API documentation
Before you begin¶
Before using Dataprep within Airflow you need to authenticate your account with TOKEN. To get connection Dataprep with Airflow you need Dataprep token. Please follow Dataprep instructions to do it.
TOKEN should be added to the Connection in Airflow in JSON format. You can check Managing Connections
The DataprepRunJobGroupOperator will run specified job. Operator required a recipe id. To identify the recipe id please use API documentation for runJobGroup E.g. if the URL is /flows/10?recipe=7, the recipe id is 7. The recipe cannot be created via this operator. It can be created only via UI which is available here. Some of parameters can be override by DAG's body request. How to do it is shown in example dag.
See following example: Set values for these fields: .. code-block:
Connection Id: "your_conn_id"
Extra: {"extra__dataprep__token": "TOKEN",
"extra__dataprep__base_url": "https://api.clouddataprep.com"}
Prerequisite Tasks¶
Run Job Group¶
Operator task is to create a job group, which launches the specified job as the authenticated user. This performs the same action as clicking on the Run Job button in the application.
To get information about jobs within a Cloud Dataprep job use:
DataprepRunJobGroupOperator
Example usage:
run_job_group = DataprepRunJobGroupOperator(task_id="run_job_group", body_request=DATA)
Get Jobs For Job Group¶
Operator task is to get information about the batch jobs within a Cloud Dataprep job.
To get information about jobs within a Cloud Dataprep job use:
DataprepGetJobsForJobGroupOperator
Example usage:
get_jobs_for_job_group = DataprepGetJobsForJobGroupOperator(
task_id="get_jobs_for_job_group", job_id=DATAPREP_JOB_ID
)
Get Job Group¶
Operator task is to get the specified job group. A job group is a job that is executed from a specific node in a flow.
To get information about jobs within a Cloud Dataprep job use:
DataprepGetJobGroupOperator
Example usage:
get_job_group = DataprepGetJobGroupOperator(
task_id="get_job_group",
job_group_id=DATAPREP_JOB_ID,
embed="",
include_deleted=False,
)