Apache Kafka Connection¶
The Apache Kafka connection type configures a connection to Apache Kafka via the
confluent-kafka Python package.
Default Connection IDs¶
Kafka hooks and operators use
kafka_default by default, this connection is very minimal and should not be assumed useful for more than the most trivial of testing.
Configuring the Connection¶
Connections are configured as a json serializable string provided to the
extra field. A full list of parameters
are described in the Confluent Kafka python library.
If you are defining the Airflow connection from the Airflow UI, the
extra field will be renamed to
Most operators and hooks will check that at the minimum the
bootstrap.servers key exists and has a value set to be valid.