airflow.providers.google.cloud.hooks.vertex_ai.batch_prediction_job
¶
This module contains a Google Cloud Vertex AI hook.
Module Contents¶
Classes¶
Hook for Google Cloud Vertex AI Batch Prediction Job APIs. |
|
Hook for Google Cloud Vertex AI Batch Prediction Job Async APIs. |
- class airflow.providers.google.cloud.hooks.vertex_ai.batch_prediction_job.BatchPredictionJobHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶
Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseHook
Hook for Google Cloud Vertex AI Batch Prediction Job APIs.
- create_batch_prediction_job(project_id, region, job_display_name, model_name, instances_format='jsonl', predictions_format='jsonl', gcs_source=None, bigquery_source=None, gcs_destination_prefix=None, bigquery_destination_prefix=None, model_parameters=None, machine_type=None, accelerator_type=None, accelerator_count=None, starting_replica_count=None, max_replica_count=None, generate_explanation=False, explanation_metadata=None, explanation_parameters=None, labels=None, encryption_spec_key_name=None, sync=True, create_request_timeout=None, batch_size=None)[source]¶
Create a batch prediction job.
- Parameters
project_id (str) – Required. Project to run training in.
region (str) – Required. Location to run training in.
job_display_name (str) – Required. The user-defined name of the BatchPredictionJob. The name can be up to 128 characters long and can be consist of any UTF-8 characters.
model_name (str | google.cloud.aiplatform.Model) – Required. A fully-qualified model resource name or model ID.
instances_format (str) – Required. The format in which instances are provided. Must be one of the formats listed in Model.supported_input_storage_formats. Default is “jsonl” when using gcs_source. If a bigquery_source is provided, this is overridden to “bigquery”.
predictions_format (str) – Required. The format in which Vertex AI outputs the predictions, must be one of the formats specified in Model.supported_output_storage_formats. Default is “jsonl” when using gcs_destination_prefix. If a bigquery_destination_prefix is provided, this is overridden to “bigquery”.
gcs_source (str | Sequence[str] | None) – Google Cloud Storage URI(-s) to your instances to run batch prediction on. They must match instances_format. May contain wildcards. For more information on wildcards, see https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNames.
bigquery_source (str | None) – BigQuery URI to a table, up to 2000 characters long. For example: bq://projectId.bqDatasetId.bqTableId
gcs_destination_prefix (str | None) – The Google Cloud Storage location of the directory where the output is to be written to. In the given directory a new directory is created. Its name is
prediction-<model-display-name>-<job-create-time>
, where timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. Inside of it filespredictions_0001.<extension>
,predictions_0002.<extension>
, …,predictions_N.<extension>
are created where<extension>
depends on chosenpredictions_format
, and N may equal 0001 and depends on the total number of successfully predicted instances. If the Model has bothinstance
andprediction
schemata defined then each such file contains predictions as per thepredictions_format
. If prediction for any instance failed (partially or completely), then an additionalerrors_0001.<extension>
,errors_0002.<extension>
,…,errors_N.<extension>
files are created (N depends on total number of failed predictions). These files contain the failed instances, as per their schema, followed by an additionalerror
field which as value has`google.rpc.Status
<Status>`__ containing onlycode
andmessage
fields.bigquery_destination_prefix (str | None) – The BigQuery project location where the output is to be written to. In the given project a new dataset is created with name
prediction_<model-display-name>_<job-create-time>
where is made BigQuery-dataset-name compatible (for example, most special characters become underscores), and timestamp is in YYYY_MM_DDThh_mm_ss_sssZ “based on ISO-8601” format. In the dataset two tables will be created,predictions
, anderrors
. If the Model has bothinstance
andprediction
schemata defined then the tables have columns as follows: Thepredictions
table contains instances for which the prediction succeeded, it has columns as per a concatenation of the Model’s instance and prediction schemata. Theerrors
table contains rows for which the prediction has failed, it has instance columns, as per the instance schema, followed by a single “errors” column, which as values has`google.rpc.Status
<Status>`__ represented as a STRUCT, and containing onlycode
andmessage
.model_parameters (dict | None) – The parameters that govern the predictions. The schema of the parameters may be specified via the Model’s parameters_schema_uri.
machine_type (str | None) – The type of machine for running batch prediction on dedicated resources. Not specifying machine type will result in batch prediction job being run with automatic resources.
accelerator_type (str | None) – The type of accelerator(s) that may be attached to the machine as per accelerator_count. Only used if machine_type is set.
accelerator_count (int | None) – The number of accelerators to attach to the machine_type. Only used if machine_type is set.
starting_replica_count (int | None) – The number of machine replicas used at the start of the batch operation. If not set, Vertex AI decides starting number, not greater than max_replica_count. Only used if machine_type is set.
max_replica_count (int | None) – The maximum number of machine replicas the batch operation may be scaled to. Only used if machine_type is set. Default is 10.
generate_explanation (bool | None) – Optional. Generate explanation along with the batch prediction results. This will cause the batch prediction output to include explanations based on the prediction_format: - bigquery: output includes a column named explanation. The value is a struct that conforms to the [aiplatform.gapic.Explanation] object. - jsonl: The JSON objects on each line include an additional entry keyed explanation. The value of the entry is a JSON object that conforms to the [aiplatform.gapic.Explanation] object. - csv: Generating explanations for CSV format is not supported.
explanation_metadata (google.cloud.aiplatform.explain.ExplanationMetadata | None) – Optional. Explanation metadata configuration for this BatchPredictionJob. Can be specified only if generate_explanation is set to True. This value overrides the value of Model.explanation_metadata. All fields of explanation_metadata are optional in the request. If a field of the explanation_metadata object is not populated, the corresponding field of the Model.explanation_metadata object is inherited. For more details, see Ref docs <http://tinyurl.com/1igh60kt>
explanation_parameters (google.cloud.aiplatform.explain.ExplanationParameters | None) – Optional. Parameters to configure explaining for Model’s predictions. Can be specified only if generate_explanation is set to True. This value overrides the value of Model.explanation_parameters. All fields of explanation_parameters are optional in the request. If a field of the explanation_parameters object is not populated, the corresponding field of the Model.explanation_parameters object is inherited. For more details, see Ref docs <http://tinyurl.com/1an4zake>
labels (dict[str, str] | None) – Optional. The labels with user-defined metadata to organize your BatchPredictionJobs. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
encryption_spec_key_name (str | None) – Optional. The Cloud KMS resource identifier of the customer managed encryption key used to protect the job. Has the form:
projects/my-project/locations/my-region/keyRings/my-kr/cryptoKeys/my-key
. The key needs to be in the same region as where the compute resource is created. If this is set, then all resources created by the BatchPredictionJob will be encrypted with the provided encryption key. Overrides encryption_spec_key_name set in aiplatform.init.sync (bool) – Whether to execute this method synchronously. If False, this method will be executed in concurrent Future and any downstream object will be immediately returned and synced when the Future has completed.
create_request_timeout (float | None) – Optional. The timeout for the create request in seconds.
batch_size (int | None) – Optional. The number of the records (e.g. instances) of the operation given in each batch to a machine replica. Machine type, and size of a single record should be considered when setting this parameter, higher value speeds up the batch operation’s execution, but too high value will result in a whole batch not fitting in a machine’s memory, and the whole operation will fail. The default value is same as in the aiplatform’s BatchPredictionJob.
- submit_batch_prediction_job(project_id, region, job_display_name, model_name, instances_format='jsonl', predictions_format='jsonl', gcs_source=None, bigquery_source=None, gcs_destination_prefix=None, bigquery_destination_prefix=None, model_parameters=None, machine_type=None, accelerator_type=None, accelerator_count=None, starting_replica_count=None, max_replica_count=None, generate_explanation=False, explanation_metadata=None, explanation_parameters=None, labels=None, encryption_spec_key_name=None, create_request_timeout=None, batch_size=None)[source]¶
Create a batch prediction job.
- Parameters
project_id (str) – Required. Project to run training in.
region (str) – Required. Location to run training in.
job_display_name (str) – Required. The user-defined name of the BatchPredictionJob. The name can be up to 128 characters long and can be consist of any UTF-8 characters.
model_name (str | google.cloud.aiplatform.Model) – Required. A fully-qualified model resource name or model ID.
instances_format (str) – Required. The format in which instances are provided. Must be one of the formats listed in Model.supported_input_storage_formats. Default is “jsonl” when using gcs_source. If a bigquery_source is provided, this is overridden to “bigquery”.
predictions_format (str) – Required. The format in which Vertex AI outputs the predictions, must be one of the formats specified in Model.supported_output_storage_formats. Default is “jsonl” when using gcs_destination_prefix. If a bigquery_destination_prefix is provided, this is overridden to “bigquery”.
gcs_source (str | Sequence[str] | None) – Google Cloud Storage URI(-s) to your instances to run batch prediction on. They must match instances_format. May contain wildcards. For more information on wildcards, see https://cloud.google.com/storage/docs/gsutil/addlhelp/WildcardNames.
bigquery_source (str | None) – BigQuery URI to a table, up to 2000 characters long. For example: bq://projectId.bqDatasetId.bqTableId
gcs_destination_prefix (str | None) – The Google Cloud Storage location of the directory where the output is to be written to. In the given directory a new directory is created. Its name is
prediction-<model-display-name>-<job-create-time>
, where timestamp is in YYYY-MM-DDThh:mm:ss.sssZ ISO-8601 format. Inside of it filespredictions_0001.<extension>
,predictions_0002.<extension>
, …,predictions_N.<extension>
are created where<extension>
depends on chosenpredictions_format
, and N may equal 0001 and depends on the total number of successfully predicted instances. If the Model has bothinstance
andprediction
schemata defined then each such file contains predictions as per thepredictions_format
. If prediction for any instance failed (partially or completely), then an additionalerrors_0001.<extension>
,errors_0002.<extension>
,…,errors_N.<extension>
files are created (N depends on total number of failed predictions). These files contain the failed instances, as per their schema, followed by an additionalerror
field which as value has`google.rpc.Status
<Status>`__ containing onlycode
andmessage
fields.bigquery_destination_prefix (str | None) – The BigQuery project location where the output is to be written to. In the given project a new dataset is created with name
prediction_<model-display-name>_<job-create-time>
where is made BigQuery-dataset-name compatible (for example, most special characters become underscores), and timestamp is in YYYY_MM_DDThh_mm_ss_sssZ “based on ISO-8601” format. In the dataset two tables will be created,predictions
, anderrors
. If the Model has bothinstance
andprediction
schemata defined then the tables have columns as follows: Thepredictions
table contains instances for which the prediction succeeded, it has columns as per a concatenation of the Model’s instance and prediction schemata. Theerrors
table contains rows for which the prediction has failed, it has instance columns, as per the instance schema, followed by a single “errors” column, which as values has`google.rpc.Status
<Status>`__ represented as a STRUCT, and containing onlycode
andmessage
.model_parameters (dict | None) – The parameters that govern the predictions. The schema of the parameters may be specified via the Model’s parameters_schema_uri.
machine_type (str | None) – The type of machine for running batch prediction on dedicated resources. Not specifying machine type will result in batch prediction job being run with automatic resources.
accelerator_type (str | None) – The type of accelerator(s) that may be attached to the machine as per accelerator_count. Only used if machine_type is set.
accelerator_count (int | None) – The number of accelerators to attach to the machine_type. Only used if machine_type is set.
starting_replica_count (int | None) – The number of machine replicas used at the start of the batch operation. If not set, Vertex AI decides starting number, not greater than max_replica_count. Only used if machine_type is set.
max_replica_count (int | None) – The maximum number of machine replicas the batch operation may be scaled to. Only used if machine_type is set. Default is 10.
generate_explanation (bool | None) – Optional. Generate explanation along with the batch prediction results. This will cause the batch prediction output to include explanations based on the prediction_format: - bigquery: output includes a column named explanation. The value is a struct that conforms to the [aiplatform.gapic.Explanation] object. - jsonl: The JSON objects on each line include an additional entry keyed explanation. The value of the entry is a JSON object that conforms to the [aiplatform.gapic.Explanation] object. - csv: Generating explanations for CSV format is not supported.
explanation_metadata (google.cloud.aiplatform.explain.ExplanationMetadata | None) – Optional. Explanation metadata configuration for this BatchPredictionJob. Can be specified only if generate_explanation is set to True. This value overrides the value of Model.explanation_metadata. All fields of explanation_metadata are optional in the request. If a field of the explanation_metadata object is not populated, the corresponding field of the Model.explanation_metadata object is inherited. For more details, see Ref docs <http://tinyurl.com/1igh60kt>
explanation_parameters (google.cloud.aiplatform.explain.ExplanationParameters | None) – Optional. Parameters to configure explaining for Model’s predictions. Can be specified only if generate_explanation is set to True. This value overrides the value of Model.explanation_parameters. All fields of explanation_parameters are optional in the request. If a field of the explanation_parameters object is not populated, the corresponding field of the Model.explanation_parameters object is inherited. For more details, see Ref docs <http://tinyurl.com/1an4zake>
labels (dict[str, str] | None) – Optional. The labels with user-defined metadata to organize your BatchPredictionJobs. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. See https://goo.gl/xmQnxf for more information and examples of labels.
encryption_spec_key_name (str | None) – Optional. The Cloud KMS resource identifier of the customer managed encryption key used to protect the job. Has the form:
projects/my-project/locations/my-region/keyRings/my-kr/cryptoKeys/my-key
. The key needs to be in the same region as where the compute resource is created. If this is set, then all resources created by the BatchPredictionJob will be encrypted with the provided encryption key. Overrides encryption_spec_key_name set in aiplatform.init.create_request_timeout (float | None) – Optional. The timeout for the create request in seconds.
batch_size (int | None) – Optional. The number of the records (e.g. instances) of the operation given in each batch to a machine replica. Machine type, and size of a single record should be considered when setting this parameter, higher value speeds up the batch operation’s execution, but too high value will result in a whole batch not fitting in a machine’s memory, and the whole operation will fail. The default value is same as in the aiplatform’s BatchPredictionJob.
- delete_batch_prediction_job(project_id, region, batch_prediction_job, retry=DEFAULT, timeout=None, metadata=())[source]¶
Delete a BatchPredictionJob. Can only be called on jobs that already finished.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
region (str) – Required. The ID of the Google Cloud region that the service belongs to.
batch_prediction_job (str) – The name of the BatchPredictionJob resource to be deleted.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
- get_batch_prediction_job(project_id, region, batch_prediction_job, retry=DEFAULT, timeout=None, metadata=())[source]¶
Get a BatchPredictionJob.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
region (str) – Required. The ID of the Google Cloud region that the service belongs to.
batch_prediction_job (str) – Required. The name of the BatchPredictionJob resource.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
- list_batch_prediction_jobs(project_id, region, filter=None, page_size=None, page_token=None, read_mask=None, retry=DEFAULT, timeout=None, metadata=())[source]¶
List BatchPredictionJobs in a Location.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
region (str) – Required. The ID of the Google Cloud region that the service belongs to.
filter (str | None) – The standard list filter. Supported fields: -
display_name
supports = and !=. -state
supports = and !=. -model_display_name
supports = and != Some examples of using the filter are: -state="JOB_STATE_SUCCEEDED" AND display_name="my_job"
-state="JOB_STATE_RUNNING" OR display_name="my_job"
-NOT display_name="my_job"
-state="JOB_STATE_FAILED"
page_size (int | None) – The standard list page size.
page_token (str | None) – The standard list page token.
read_mask (str | None) – Mask specifying which fields to read.
retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.
- class airflow.providers.google.cloud.hooks.vertex_ai.batch_prediction_job.BatchPredictionJobAsyncHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶
Bases:
airflow.providers.google.common.hooks.base_google.GoogleBaseAsyncHook
Hook for Google Cloud Vertex AI Batch Prediction Job Async APIs.
- async get_batch_prediction_job(project_id, location, job_id, retry=DEFAULT, timeout=None, metadata=())[source]¶
Retrieve a batch prediction tuning job.
- Parameters
project_id (str) – Required. The ID of the Google Cloud project that the job belongs to.
location (str) – Required. The ID of the Google Cloud region that the job belongs to.
job_id (str) – Required. The hyperparameter tuning job id.
retry (google.api_core.retry.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.
timeout (float | None) – The timeout for this request.
metadata (Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.