airflow.providers.google.cloud.operators.dataform
¶
Module Contents¶
Classes¶
Creates a new CompilationResult in a given project and location. |
|
Fetches a single CompilationResult. |
|
Creates a new WorkflowInvocation in a given Repository. |
|
Fetches a single WorkflowInvocation. |
|
Requests cancellation of a running WorkflowInvocation. |
|
Creates repository. |
|
Deletes repository. |
|
Creates workspace. |
|
Deletes workspace. |
|
Writes new file to specified workspace. |
|
Makes new directory in specified workspace. |
|
Removes file in specified workspace. |
|
Removes directory in specified workspace. |
|
Install NPM dependencies in the provided workspace. |
- class airflow.providers.google.cloud.operators.dataform.DataformCreateCompilationResultOperator(project_id, region, repository_id, compilation_result, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a new CompilationResult in a given project and location.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.
compilation_result (google.cloud.dataform_v1beta1.types.CompilationResult | dict) – Required. The compilation result to create.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformGetCompilationResultOperator(project_id, region, repository_id, compilation_result_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Fetches a single CompilationResult.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.
compilation_result_id (str) – The Id of the Dataform Compilation Result
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformCreateWorkflowInvocationOperator(project_id, region, repository_id, workflow_invocation, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, asynchronous=False, wait_time=10, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates a new WorkflowInvocation in a given Repository.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.
workflow_invocation (google.cloud.dataform_v1beta1.types.WorkflowInvocation | dict) – Required. The workflow invocation resource to create.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (int | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
asynchronous (bool) – Flag to return workflow_invocation_id from the Dataform API. This is useful for submitting long-running workflows and waiting on them asynchronously using the DataformWorkflowInvocationStateSensor
wait_time (int) – Number of seconds between checks
- class airflow.providers.google.cloud.operators.dataform.DataformGetWorkflowInvocationOperator(project_id, region, repository_id, workflow_invocation_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Fetches a single WorkflowInvocation.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.
workflow_invocation_id (str) – the workflow invocation resource’s id.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformCancelWorkflowInvocationOperator(project_id, region, repository_id, workflow_invocation_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Requests cancellation of a running WorkflowInvocation.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.
workflow_invocation_id (str) – the workflow invocation resource’s id.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformCreateRepositoryOperator(project_id, region, repository_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates repository.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the task belongs to.
region (str) – Required. The ID of the Google Cloud region that the task belongs to.
repository_id (str) – Required. The ID of the Dataform repository that the task belongs to.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformDeleteRepositoryOperator(project_id, region, repository_id, force=True, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Deletes repository.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where repository located.
region (str) – Required. The ID of the Google Cloud region where repository located.
repository_id (str) – Required. The ID of the Dataform repository that should be deleted.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformCreateWorkspaceOperator(project_id, region, repository_id, workspace_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Creates workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace should be in.
region (str) – Required. Name of the Google Cloud region that where workspace should be in.
repository_id (str) – Required. The ID of the Dataform repository that the workspace belongs to.
workspace_id (str) – Required. The ID of the new workspace that will be created.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformDeleteWorkspaceOperator(project_id, region, repository_id, workspace_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Deletes workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace located.
region (str) – Required. The ID of the Google Cloud region where workspace located.
repository_id (str) – Required. The ID of the Dataform repository where workspace located.
workspace_id (str) – Required. The ID of the Dataform workspace that should be deleted.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformWriteFileOperator(project_id, region, repository_id, workspace_id, filepath, contents, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Writes new file to specified workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace located.
region (str) – Required. The ID of the Google Cloud region where workspace located.
repository_id (str) – Required. The ID of the Dataform repository where workspace located.
workspace_id (str) – Required. The ID of the Dataform workspace where files should be created.
filepath (str) – Required. Path to file including name of the file relative to workspace root.
contents (bytes) – Required. Content of the file to be written.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformMakeDirectoryOperator(project_id, region, repository_id, workspace_id, directory_path, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Makes new directory in specified workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace located.
region (str) – Required. The ID of the Google Cloud region where workspace located.
repository_id (str) – Required. The ID of the Dataform repository where workspace located.
workspace_id (str) – Required. The ID of the Dataform workspace where directory should be created.
path – Required. The directory’s full path including directory name, relative to the workspace root.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformRemoveFileOperator(project_id, region, repository_id, workspace_id, filepath, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Removes file in specified workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace located.
region (str) – Required. The ID of the Google Cloud region where workspace located.
repository_id (str) – Required. The ID of the Dataform repository where workspace located.
workspace_id (str) – Required. The ID of the Dataform workspace where directory located.
filepath (str) – Required. The full path including name of the file, relative to the workspace root.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformRemoveDirectoryOperator(project_id, region, repository_id, workspace_id, directory_path, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Removes directory in specified workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace located.
region (str) – Required. The ID of the Google Cloud region where workspace located.
repository_id (str) – Required. The ID of the Dataform repository where workspace located.
workspace_id (str) – Required. The ID of the Dataform workspace where directory located.
path – Required. The directory’s full path including directory name, relative to the workspace root.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).
- class airflow.providers.google.cloud.operators.dataform.DataformInstallNpmPackagesOperator(project_id, region, repository_id, workspace_id, retry=DEFAULT, timeout=None, metadata=(), gcp_conn_id='google_cloud_default', impersonation_chain=None, *args, **kwargs)[source]¶
Bases:
airflow.providers.google.cloud.operators.cloud_base.GoogleCloudBaseOperator
Install NPM dependencies in the provided workspace.
Requires “package.json” to be created in the workspace.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project where workspace located.
region (str) – Required. The ID of the Google Cloud region where workspace located.
repository_id (str) – Required. The ID of the Dataform repository where workspace located.
workspace_id (str) – Required. The ID of the Dataform workspace.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account (templated).