Apache Kafka

apache-airflow-providers-apache-kafka

Apache Kafka

Works with Airflow 2.11+
1.5M
Monthly Downloads
10
Total Modules
View Docs Source
Install:
pip install apache-airflow-providers-apache-kafka==1.13.0

Airflow

2.11+

Python

>=3.10

Dependencies (4)

Show all Hide apache-airflow>=2.11.0 apache-airflow-providers-common-compat>=1.12.0 asgiref>=2.3.0 confluent-kafka>=2.6.0

Connections (1)

Modules

O

ConsumeFromTopicOperator

An operator that consumes from Kafka a topic(s) and processing the messages.

airflow.providers.apache.kafka.operators.consume.ConsumeFromTopicOperator
View Docs Source
O

ProduceToTopicOperator

An operator that produces messages to a Kafka topic.

airflow.providers.apache.kafka.operators.produce.ProduceToTopicOperator
View Docs Source
H

KafkaBaseHook

A base hook for interacting with Apache Kafka.

airflow.providers.apache.kafka.hooks.base.KafkaBaseHook
View Docs Source
H

KafkaAdminClientHook

A hook for interacting with the Kafka Cluster.

airflow.providers.apache.kafka.hooks.client.KafkaAdminClientHook
View Docs Source
H

KafkaConsumerHook

A hook for creating a Kafka Consumer.

airflow.providers.apache.kafka.hooks.consume.KafkaConsumerHook
View Docs Source
H

KafkaProducerHook

A hook for creating a Kafka Producer.

airflow.providers.apache.kafka.hooks.produce.KafkaProducerHook
View Docs Source
S

AwaitMessageSensor

An Airflow sensor that defers until a specific message is published to Kafka.

airflow.providers.apache.kafka.sensors.kafka.AwaitMessageSensor
View Docs Source
S

AwaitMessageTriggerFunctionSensor

Defer until a specific message is published to Kafka, trigger a registered function, then resume waiting.

airflow.providers.apache.kafka.sensors.kafka.AwaitMessageTriggerFunctionSensor
View Docs Source
T

AwaitMessageTrigger

A trigger that waits for a message matching specific criteria to arrive in Kafka.

airflow.providers.apache.kafka.triggers.await_message.AwaitMessageTrigger
View Docs Source
T

KafkaMessageQueueTrigger

A dedicated trigger for Kafka message queues that extends ``MessageQueueTrigger``.

airflow.providers.apache.kafka.triggers.msg_queue.KafkaMessageQueueTrigger
View Docs Source