airflow.providers.apache.kafka.triggers.await_message
¶
Module Contents¶
Classes¶
A trigger that waits for a message matching specific criteria to arrive in Kafka. |
- class airflow.providers.apache.kafka.triggers.await_message.AwaitMessageTrigger(topics, apply_function, kafka_config_id='kafka_default', apply_function_args=None, apply_function_kwargs=None, poll_timeout=1, poll_interval=5)[source]¶
Bases:
airflow.triggers.base.BaseTrigger
A trigger that waits for a message matching specific criteria to arrive in Kafka.
The behavior of the consumer of this trigger is as follows: - poll the Kafka topics for a message, if no message returned, sleep - process the message with provided callable and commit the message offset:
if callable returns any data, raise a TriggerEvent with the return data
else continue to next message
- Parameters
kafka_config_id (str) – The connection object to use, defaults to “kafka_default”
topics (Sequence[str]) – The topic (or topic regex) that should be searched for messages
apply_function (str) – the location of the function to apply to messages for determination of matching criteria. (In python dot notation as a string)
apply_function_args (Sequence[Any] | None) – A set of arguments to apply to the callable, defaults to None
apply_function_kwargs (dict[Any, Any] | None) – A set of key word arguments to apply to the callable, defaults to None, defaults to None
poll_timeout (float) – How long the Kafka client should wait before returning from a poll request to Kafka (seconds), defaults to 1
poll_interval (float) – How long the trigger should sleep after reaching the end of the Kafka log (seconds), defaults to 5
- async run()[source]¶
Run the trigger in an asynchronous context.
The trigger should yield an Event whenever it wants to fire off an event, and return None if it is finished. Single-event triggers should thus yield and then immediately return.
If it yields, it is likely that it will be resumed very quickly, but it may not be (e.g. if the workload is being moved to another triggerer process, or a multi-event trigger was being used for a single-event task defer).
In either case, Trigger classes should assume they will be persisted, and then rely on cleanup() being called when they are no longer needed.