airflow.providers.apache.kafka.operators.produce

Module Contents

Classes

ProduceToTopicOperator

An operator that produces messages to a Kafka topic.

Functions

acked(err, msg)

Attributes

local_logger

airflow.providers.apache.kafka.operators.produce.local_logger[source]
airflow.providers.apache.kafka.operators.produce.acked(err, msg)[source]
class airflow.providers.apache.kafka.operators.produce.ProduceToTopicOperator(topic, producer_function, kafka_config_id='kafka_default', producer_function_args=None, producer_function_kwargs=None, delivery_callback=None, synchronous=True, poll_timeout=0, **kwargs)[source]

Bases: airflow.models.BaseOperator

An operator that produces messages to a Kafka topic.

Registers a producer to a kafka topic and publishes messages to the log.

Parameters

  • kafka_config_id (str) – The connection object to use, defaults to “kafka_default”

  • topic (str) – The topic the producer should produce to, defaults to None

  • producer_function (str | Callable[Ellipsis, Any]) – The function that generates key/value pairs as messages for production, defaults to None

  • producer_function_args (Sequence[Any] | None) – Additional arguments to be applied to the producer callable, defaults to None

  • producer_function_kwargs (dict[Any, Any] | None) – Additional keyword arguments to be applied to the producer callable, defaults to None

  • delivery_callback (str | None) – The callback to apply after delivery(or failure) of a message, defaults to None

  • synchronous (bool) – If writes to kafka should be fully synchronous, defaults to True

  • poll_timeout (float) – How long of a delay should be applied when calling poll after production to kafka, defaults to 0

Raises

AirflowException – _description_

See also

For more information on how to use this operator, take a look at the guide: ProduceToTopicOperator

template_fields = ('topic', 'producer_function', 'producer_function_args', 'producer_function_kwargs', 'kafka_config_id')[source]
execute(context)[source]

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

Was this entry helpful?