airflow.providers.amazon.aws.triggers.s3

Module Contents

Classes

S3KeyTrigger

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker.

S3KeysUnchangedTrigger

S3KeysUnchangedTrigger is fired as deferred class with params to run the task in trigger worker.
class airflow.providers.amazon.aws.triggers.s3.S3KeyTrigger(bucket_name, bucket_key, wildcard_match=False, aws_conn_id='aws_default', poke_interval=5.0, should_check_fn=False, use_regex=False, **hook_params)[source]

Bases: airflow.triggers.base.BaseTrigger

S3KeyTrigger is fired as deferred class with params to run the task in trigger worker.

Parameters

  • bucket_name (str) – Name of the S3 bucket. Only needed when bucket_key is not provided as a full s3:// url.

  • bucket_key (str | list[str]) – The key being waited on. Supports full s3:// style url or relative path from root level. When it’s specified as a full s3:// url, please leave bucket_name as None.

  • wildcard_match (bool) – whether the bucket_key should be interpreted as a Unix wildcard pattern

  • aws_conn_id (str) – reference to the s3 connection

  • use_regex (bool) – whether to use regex to check bucket

  • hook_params (Any) – params for hook its optional

serialize()[source]

Serialize S3KeyTrigger arguments and classpath.

hook()[source]
async run()[source]

Make an asynchronous connection using S3HookAsync.

class airflow.providers.amazon.aws.triggers.s3.S3KeysUnchangedTrigger(bucket_name, prefix, inactivity_period=60 * 60, min_objects=1, inactivity_seconds=0, previous_objects=None, allow_delete=True, aws_conn_id='aws_default', last_activity_time=None, verify=None, **hook_params)[source]

Bases: airflow.triggers.base.BaseTrigger

S3KeysUnchangedTrigger is fired as deferred class with params to run the task in trigger worker.

Parameters

  • bucket_name (str) – Name of the S3 bucket. Only needed when bucket_key is not provided as a full s3:// url.

  • prefix (str) – The prefix being waited on. Relative path from bucket root level.

  • inactivity_period (float) – The total seconds of inactivity to designate keys unchanged. Note, this mechanism is not real time and this operator may not return until a poke_interval after this period has passed with no additional objects sensed.

  • min_objects (int) – The minimum number of objects needed for keys unchanged sensor to be considered valid.

  • inactivity_seconds (int) – reference to the seconds of inactivity

  • previous_objects (set[str] | None) – The set of object ids found during the last poke.

  • allow_delete (bool) – Should this sensor consider objects being deleted

  • aws_conn_id (str) – reference to the s3 connection

  • last_activity_time (datetime.datetime | None) – last modified or last active time

  • verify (bool | str | None) – Whether or not to verify SSL certificates for S3 connection. By default SSL certificates are verified.

  • hook_params (Any) – params for hook its optional

serialize()[source]

Serialize S3KeysUnchangedTrigger arguments and classpath.

hook()[source]
async run()[source]

Make an asynchronous connection using S3Hook.

Was this entry helpful?