`airflow.providers.amazon.aws.log.s3_task_handler`¶

Module Contents¶

Classes¶

S3TaskHandler

S3TaskHandler is a python log handler that handles and reads

Functions¶

get_default_delete_local_copy()

Load delete_local_logs conf if Airflow version > 2.6 and return False if not.

airflow.providers.amazon.aws.log.s3_task_handler.get_default_delete_local_copy()[source]¶

Load delete_local_logs conf if Airflow version > 2.6 and return False if not.

TODO: delete this function when min airflow version >= 2.6

class airflow.providers.amazon.aws.log.s3_task_handler.S3TaskHandler(base_log_folder, s3_log_folder, filename_template=None, **kwargs)[source]¶

Bases: airflow.utils.log.file_task_handler.FileTaskHandler, airflow.utils.log.logging_mixin.LoggingMixin

S3TaskHandler is a python log handler that handles and reads task instance logs. It extends airflow FileTaskHandler and uploads to and reads from S3 remote storage.

trigger_should_wrap = True[source]¶

hook()[source]¶: Returns S3Hook.

set_context(ti)[source]¶

Provide task_instance context to airflow task handler.

Generally speaking returns None. But if attr maintain_propagate has been set to propagate, then returns sentinel MAINTAIN_PROPAGATE. This has the effect of overriding the default behavior to set propagate to False whenever set_context is called. At time of writing, this functionality is only used in unit testing.

Parameters: ti – task instance object

close()[source]¶: Close and upload local log file to remote storage S3.

s3_log_exists(remote_log_location)[source]¶

Check if remote_log_location exists in remote storage.

Parameters: remote_log_location (str) – log’s location in remote storage
Returns: True if location exists else False
Return type: bool

s3_read(remote_log_location, return_error=False)[source]¶

Returns the log found at the remote_log_location. Returns ‘’ if no logs are found or there is an error.

Parameters

remote_log_location (str) – the log’s location in remote storage
return_error (bool) – if True, returns a string error message if an error occurs. Otherwise returns ‘’ when an error occurs.

Returns

the log found at the remote_log_location

Return type

str

s3_write(log, remote_log_location, append=True, max_retry=1)[source]¶

Writes the log to the remote_log_location and return True when done. Fails silently: and return False if no log was created.