airflow.providers.google.cloud.hooks.mlengine

This module contains a Google ML Engine Hook.

Module Contents

airflow.providers.google.cloud.hooks.mlengine.log[source]
airflow.providers.google.cloud.hooks.mlengine._AIRFLOW_VERSION[source]
airflow.providers.google.cloud.hooks.mlengine._poll_with_exponential_delay(request, execute_num_retries, max_n, is_done_func, is_error_func)[source]
Execute request with exponential delay.

This method is intended to handle and retry in case of api-specific errors, such as 429 “Too Many Requests”, unlike the request.execute which handles lower level errors like ConnectionError/socket.timeout/ssl.SSLError.

Parameters

  • request (googleapiclient.http.HttpRequest) – request to be executed.

  • execute_num_retries (int) – num_retries for request.execute method.

  • max_n (int) – number of times to retry request in this method.

  • is_done_func (callable) – callable to determine if operation is done.

  • is_error_func (callable) – callable to determine if operation is failed.

Returns

response

Return type

httplib2.Response

class airflow.providers.google.cloud.hooks.mlengine.MLEngineHook[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google ML Engine APIs.

All the methods in the hook where project_id is used must be called with keyword arguments rather than positional.

get_conn(self)[source]

Retrieves the connection to MLEngine.

Returns

Google MLEngine services object.

create_job(self, job: dict, project_id: str, use_existing_job_fn: Optional[Callable] = None)[source]

Launches a MLEngine job and wait for it to reach a terminal state.

Parameters

  • project_id (str) – The Google Cloud project id within which MLEngine job will be launched. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • job (dict) –

    MLEngine Job object that should be provided to the MLEngine API, such as:

    {
  'jobId': 'my_job_id',
  'trainingInput': {
    'scaleTier': 'STANDARD_1',
    ...
  }
}

  • use_existing_job_fn (function) – In case that a MLEngine job with the same job_id already exist, this method (if provided) will decide whether we should use this existing job, continue waiting for it to finish and returning the job object. It should accepts a MLEngine job object, and returns a boolean value indicating whether it is OK to reuse the existing job. If ‘use_existing_job_fn’ is not provided, we by default reuse the existing MLEngine job.

Returns

The MLEngine job object if the job successfully reach a terminal state (which might be FAILED or CANCELLED state).

Return type

dict

cancel_job(self, job_id: str, project_id: str)[source]

Cancels a MLEngine job.

Parameters

  • project_id (str) – The Google Cloud project id within which MLEngine job will be cancelled. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • job_id (str) – A unique id for the want-to-be cancelled Google MLEngine training job.

Returns

Empty dict if cancelled successfully

Return type

dict

Raises

googleapiclient.errors.HttpError

_get_job(self, project_id: str, job_id: str)[source]

Gets a MLEngine job based on the job id.

Parameters

  • project_id (str) – The project in which the Job is located. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

  • job_id (str) – A unique id for the Google MLEngine job. (templated)

Returns

MLEngine job object if succeed.

Return type

dict

Raises

googleapiclient.errors.HttpError

_wait_for_job_done(self, project_id: str, job_id: str, interval: int = 30)[source]

Waits for the Job to reach a terminal state.

This method will periodically check the job state until the job reach a terminal state.

Parameters

  • project_id (str) – The project in which the Job is located. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

  • job_id (str) – A unique id for the Google MLEngine job. (templated)

  • interval (int) – Time expressed in seconds after which the job status is checked again. (templated)

Raises

googleapiclient.errors.HttpError

create_version(self, model_name: str, version_spec: Dict, project_id: str)[source]

Creates the Version on Google Cloud ML Engine.

Parameters

  • version_spec (dict) – A dictionary containing the information about the version. (templated)

  • model_name (str) – The name of the Google Cloud ML Engine model that the version belongs to. (templated)

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

Returns

If the version was created successfully, returns the operation. Otherwise raises an error .

Return type

dict

set_default_version(self, model_name: str, version_name: str, project_id: str)[source]

Sets a version to be the default. Blocks until finished.

Parameters

  • model_name (str) – The name of the Google Cloud ML Engine model that the version belongs to. (templated)

  • version_name (str) – A name to use for the version being operated upon. (templated)

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

Returns

If successful, return an instance of Version. Otherwise raises an error.

Return type

dict

Raises

googleapiclient.errors.HttpError

list_versions(self, model_name: str, project_id: str)[source]

Lists all available versions of a model. Blocks until finished.

Parameters

  • model_name (str) – The name of the Google Cloud ML Engine model that the version belongs to. (templated)

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

Returns

return an list of instance of Version.

Return type

List[Dict]

Raises

googleapiclient.errors.HttpError

delete_version(self, model_name: str, version_name: str, project_id: str)[source]

Deletes the given version of a model. Blocks until finished.

Parameters

  • model_name (str) – The name of the Google Cloud ML Engine model that the version belongs to. (templated)

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs.

Returns

If the version was deleted successfully, returns the operation. Otherwise raises an error.

Return type

Dict

create_model(self, model: dict, project_id: str)[source]

Create a Model. Blocks until finished.

Parameters

  • model (dict) – A dictionary containing the information about the model.

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

Returns

If the version was created successfully, returns the instance of Model. Otherwise raises an error.

Return type

Dict

Raises

googleapiclient.errors.HttpError

get_model(self, model_name: str, project_id: str)[source]

Gets a Model. Blocks until finished.

Parameters

  • model_name (str) – The name of the model.

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

Returns

If the model exists, returns the instance of Model. Otherwise return None.

Return type

Dict

Raises

googleapiclient.errors.HttpError

delete_model(self, model_name: str, project_id: str, delete_contents: bool = False)[source]

Delete a Model. Blocks until finished.

Parameters

  • model_name (str) – The name of the model.

  • delete_contents (bool) – Whether to force the deletion even if the models is not empty. Will delete all version (if any) in the dataset if set to True. The default value is False.

  • project_id (str) – The Google Cloud project name to which MLEngine model belongs. If set to None or missing, the default project_id from the Google Cloud connection is used. (templated)

Raises

googleapiclient.errors.HttpError

_delete_all_versions(self, model_name: str, project_id: str)[source]
_append_label(self, model: dict)[source]

Was this entry helpful?