airflow.providers.apache.kafka.triggers.await_message

Classes

AwaitMessageTrigger

A trigger that waits for a message matching specific criteria to arrive in Kafka.

Module Contents

class airflow.providers.apache.kafka.triggers.await_message.AwaitMessageTrigger(topics, apply_function, kafka_config_id='kafka_default', apply_function_args=None, apply_function_kwargs=None, poll_timeout=1, poll_interval=5)[source]

Bases: airflow.triggers.base.BaseTrigger

A trigger that waits for a message matching specific criteria to arrive in Kafka.

The behavior of the consumer of this trigger is as follows: - poll the Kafka topics for a message, if no message returned, sleep - process the message with provided callable and commit the message offset:

  • if callable returns any data, raise a TriggerEvent with the return data

  • else continue to next message

Parameters:

  • kafka_config_id (str) – The connection object to use, defaults to “kafka_default”

  • topics (collections.abc.Sequence[str]) – The topic (or topic regex) that should be searched for messages

  • apply_function (str) – the location of the function to apply to messages for determination of matching criteria. (In python dot notation as a string)

  • apply_function_args (collections.abc.Sequence[Any] | None) – A set of arguments to apply to the callable, defaults to None

  • apply_function_kwargs (dict[Any, Any] | None) – A set of key word arguments to apply to the callable, defaults to None, defaults to None

  • poll_timeout (float) – How long the Kafka client should wait before returning from a poll request to Kafka (seconds), defaults to 1

  • poll_interval (float) – How long the trigger should sleep after reaching the end of the Kafka log (seconds), defaults to 5

topics[source]
apply_function[source]
apply_function_args = ()[source]
apply_function_kwargs[source]
kafka_config_id = 'kafka_default'[source]
poll_timeout = 1[source]
poll_interval = 5[source]
serialize()[source]

Return the information needed to reconstruct this Trigger.

Returns:

Tuple of (class path, keyword arguments needed to re-instantiate).

Return type:

tuple[str, dict[str, Any]]

async run()[source]

Run the trigger in an asynchronous context.

The trigger should yield an Event whenever it wants to fire off an event, and return None if it is finished. Single-event triggers should thus yield and then immediately return.

If it yields, it is likely that it will be resumed very quickly, but it may not be (e.g. if the workload is being moved to another triggerer process, or a multi-event trigger was being used for a single-event task defer).

In either case, Trigger classes should assume they will be persisted, and then rely on cleanup() being called when they are no longer needed.

Was this entry helpful?