airflow.providers.google.cloud.log.stackdriver_task_handler

Handler that integrates with Stackdriver

Module Contents

airflow.providers.google.cloud.log.stackdriver_task_handler.DEFAULT_LOGGER_NAME = airflow[source]
airflow.providers.google.cloud.log.stackdriver_task_handler._GLOBAL_RESOURCE[source]
airflow.providers.google.cloud.log.stackdriver_task_handler._DEFAULT_SCOPESS[source]
class airflow.providers.google.cloud.log.stackdriver_task_handler.StackdriverTaskHandler(gcp_key_path: Optional[str] = None, scopes: Optional[Collection[str]] = _DEFAULT_SCOPESS, name: str = DEFAULT_LOGGER_NAME, transport: Type[Transport] = BackgroundThreadTransport, resource: Resource = _GLOBAL_RESOURCE, labels: Optional[Dict[str, str]] = None)[source]

Bases: logging.Handler

Handler that directly makes Stackdriver logging API calls.

This is a Python standard logging handler using that can be used to route Python standard logging messages directly to the Stackdriver Logging API.

It can also be used to save logs for executing tasks. To do this, you should set as a handler with the name “tasks”. In this case, it will also be used to read the log for display in Web UI.

This handler supports both an asynchronous and synchronous transport.

Parameters

  • gcp_key_path (str) – Path to Google Cloud Credential JSON file. If omitted, authorization based on the Application Default Credentials will be used.

  • scopes (Sequence[str]) – OAuth scopes for the credentials,

  • name (str) – the name of the custom log in Stackdriver Logging. Defaults to ‘airflow’. The name of the Python logger will be represented in the python_logger field.

  • transport (type) – Class for creating new transport objects. It should extend from the base google.cloud.logging.handlers.Transport type and implement :meth`google.cloud.logging.handlers.Transport.send`. Defaults to google.cloud.logging.handlers.BackgroundThreadTransport. The other option is google.cloud.logging.handlers.SyncTransport.

  • resource (Resource) – (Optional) Monitored resource of the entry, defaults to the global resource type.

  • labels (dict) – (Optional) Mapping of labels for the entry.

LABEL_TASK_ID = task_id[source]
LABEL_DAG_ID = dag_id[source]
LABEL_EXECUTION_DATE = execution_date[source]
LABEL_TRY_NUMBER = try_number[source]
LOG_VIEWER_BASE_URL = https://console.cloud.google.com/logs/viewer[source]
LOG_NAME = Google Stackdriver[source]
_client[source]

The Cloud Library API client

_logging_service_client[source]

The Cloud logging service v2 client.

log_name[source]

Return log name.

_credentials_and_project(self)[source]
_transport(self)[source]

Object responsible for sending data to Stackdriver

emit(self, record: logging.LogRecord)[source]

Actually log the specified logging record.

Parameters

record (logging.LogRecord) – The record to be logged.

set_context(self, task_instance: TaskInstance)[source]

Configures the logger to add information with information about the current task

Parameters

task_instance (airflow.models.TaskInstance) – Currently executed task

read(self, task_instance: TaskInstance, try_number: Optional[int] = None, metadata: Optional[Dict] = None)[source]

Read logs of given task instance from Stackdriver logging.

Parameters

  • task_instance (airflow.models.TaskInstance) – task instance object

  • try_number (Optional[int]) – task instance try_number to read logs from. If None it returns all logs

  • metadata (Dict) – log metadata. It is used for steaming log reading and auto-tailing.

Returns

a tuple of ( list of (one element tuple with two element tuple - hostname and logs) and list of metadata)

Return type

Tuple[List[Tuple[Tuple[str, str]]], List[Dict[str, str]]]

_prepare_log_filter(self, ti_labels: Dict[str, str])[source]

Prepares the filter that chooses which log entries to fetch.

More information: https://cloud.google.com/logging/docs/reference/v2/rest/v2/entries/list#body.request_body.FIELDS.filter https://cloud.google.com/logging/docs/view/advanced-queries

Parameters

ti_labels – Task Instance’s labels that will be used to search for logs

Type

Dict[str, str]

Returns

logs filter

_read_logs(self, log_filter: str, next_page_token: Optional[str], all_pages: bool)[source]

Sends requests to the Stackdriver service and downloads logs.

Parameters

  • log_filter (str) – Filter specifying the logs to be downloaded.

  • next_page_token – The token of the page from which the log download will start. If None is passed, it will start from the first page.

  • all_pages – If True is passed, all subpages will be downloaded. Otherwise, only the first page will be downloaded

Returns

A token that contains the following items: * string with logs * Boolean value describing whether there are more logs, * token of the next page

Return type

Tuple[str, bool, str]

_read_single_logs_page(self, log_filter: str, page_token: Optional[str] = None)[source]

Sends requests to the Stackdriver service and downloads single pages with logs.

Parameters

  • log_filter (str) – Filter specifying the logs to be downloaded.

  • page_token (str) – The token of the page to be downloaded. If None is passed, the first page will be downloaded.

Returns

Downloaded logs and next page token

Return type

Tuple[str, str]

classmethod _task_instance_to_labels(cls, ti: TaskInstance)[source]
_resource_path(self)[source]
get_external_log_url(self, task_instance: TaskInstance, try_number: int)[source]

Creates an address for an external log collecting service. :param task_instance: task instance object :type: task_instance: TaskInstance :param try_number: task instance try_number to read logs from. :type try_number: Optional[int] :return: URL to the external log collection service :rtype: str

close(self)[source]

Was this entry helpful?