airflow.providers.apache.hdfs.log.hdfs_task_handler

Module Contents

Classes

HdfsTaskHandler

Logging handler to upload and read from HDFS.
class airflow.providers.apache.hdfs.log.hdfs_task_handler.HdfsTaskHandler(base_log_folder, hdfs_log_folder, filename_template=None, **kwargs)[source]

Bases: airflow.utils.log.file_task_handler.FileTaskHandler, airflow.utils.log.logging_mixin.LoggingMixin

Logging handler to upload and read from HDFS.

hook()[source]

Returns WebHDFSHook.

set_context(ti)[source]

Provide task_instance context to airflow task handler.

Generally speaking returns None. But if attr maintain_propagate has been set to propagate, then returns sentinel MAINTAIN_PROPAGATE. This has the effect of overriding the default behavior to set propagate to False whenever set_context is called. At time of writing, this functionality is only used in unit testing.

Parameters

  • ti – task instance object

  • identifier – if set, adds suffix to log file. For use when relaying exceptional messages to task logs from a context other than task or trigger run

close()[source]

Close and upload local log file to HDFS.

Was this entry helpful?