Common AI

incubation

apache-airflow-providers-common-ai

AI/LLM hooks and operators for Airflow pipelines using pydantic-ai.

Works with Airflow 3.0+

Install:

pip install apache-airflow-providers-common-ai==0.10.0

Extras:

Version:

Airflow

3.0+

Python

>=3.10

Dependencies (4)

Show all Hide

apache-airflow>=3.0.0 apache-airflow-providers-common-compat>=1.15.0 apache-airflow-providers-standard>=1.20.0 pydantic-ai-slim>=2.33.0

Connections (7)

Connection type	External services (examples)
`pydanticai`	OpenAI, Anthropic, Google, AWS Bedrock, Groq, Mistral AI, DeepSeek, Ollama, vLLM
`pydanticai_azure`	Azure OpenAI
`pydanticai_bedrock`	AWS Bedrock
`pydanticai_vertex`	Google Vertex AI
`langchain`	OpenAI, Anthropic, Groq, Mistral AI, DeepSeek, Ollama, vLLM
`llamaindex`	OpenAI

Modules

Durable (includes deferrable) (4) Deferrable only (2)

AgentOperator Durable

Run a pydantic-ai Agent with tools and multi-turn reasoning.

airflow.providers.common.ai.operators.agent.AgentOperator

View Docs Source

LLMOperator

Call an LLM with a prompt and return the output.

airflow.providers.common.ai.operators.llm.LLMOperator

View Docs Source

LLMBatchOperator Deferrable

Submit prompts as a provider batch job and land results on object storage.

airflow.providers.common.ai.operators.llm_batch.LLMBatchOperator

View Docs Source

LLMFileAnalysisOperator

Analyze files from object storage or local storage using a single LLM call.

airflow.providers.common.ai.operators.llm_file_analysis.LLMFileAnalysisOperator

View Docs Source

LLMBranchOperator

Ask an LLM to choose which downstream task(s) to execute.

airflow.providers.common.ai.operators.llm_branch.LLMBranchOperator

View Docs Source

LLMSQLQueryOperator

Generate SQL queries from natural language using an LLM.

airflow.providers.common.ai.operators.llm_sql.LLMSQLQueryOperator

View Docs Source

LLMSchemaCompareOperator

Compare schemas across different database systems and detect drift using LLM reasoning.

airflow.providers.common.ai.operators.llm_schema_compare.LLMSchemaCompareOperator

View Docs Source

DocumentLoaderOperator

Parse files into ``list[dict(text, metadata)]`` for downstream embedding.

airflow.providers.common.ai.operators.document_loader.DocumentLoaderOperator

View Docs Source

LlamaIndexEmbeddingOperator

Chunk documents and produce embedding vectors using LlamaIndex.

airflow.providers.common.ai.operators.llamaindex_embedding.LlamaIndexEmbeddingOperator

View Docs Source

LlamaIndexRetrievalOperator

Retrieve relevant document chunks from a persisted LlamaIndex index.

airflow.providers.common.ai.operators.llamaindex_retrieval.LlamaIndexRetrievalOperator

View Docs Source

PydanticAIAzureHook

Hook for Azure OpenAI via pydantic-ai.

airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIAzureHook

View Docs Source

PydanticAIBedrockHook

Hook for AWS Bedrock via pydantic-ai.

airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIBedrockHook

View Docs Source

PydanticAIHook

Hook for LLM access via pydantic-ai.

airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIHook

View Docs Source

PydanticAIVertexHook

Hook for Google Vertex AI (or Generative Language API) via pydantic-ai.

airflow.providers.common.ai.hooks.pydantic_ai.PydanticAIVertexHook

View Docs Source

MCPHook

Hook for connecting to MCP (Model Context Protocol) servers.

airflow.providers.common.ai.hooks.mcp.MCPHook

View Docs Source

LangChainHook

Bridge an Airflow connection to LangChain chat and embedding models.

airflow.providers.common.ai.hooks.langchain.LangChainHook

View Docs Source

LlamaIndexHook

Bridge an Airflow connection to LlamaIndex chat and embedding models.

airflow.providers.common.ai.hooks.llamaindex.LlamaIndexHook

View Docs Source

LLMBatchTrigger

Poll a batch adapter until the batch reaches a terminal state.

airflow.providers.common.ai.triggers.llm_batch.LLMBatchTrigger

View Docs Source

HookToolset

Expose selected methods of an Airflow Hook as pydantic-ai tools.

airflow.providers.common.ai.toolsets.hook.HookToolset

View Docs Source

SQLToolset

Curated toolset that gives an LLM agent safe access to a SQL database.

airflow.providers.common.ai.toolsets.sql.SQLToolset

View Docs Source

DataFusionToolset

Curated toolset that gives an LLM agent SQL access to object-storage data via Apache DataFusion.

airflow.providers.common.ai.toolsets.datafusion.DataFusionToolset

View Docs Source

LoggingToolset

Wrap a toolset to log each tool call with timing.

airflow.providers.common.ai.toolsets.logging.LoggingToolset

View Docs Source

MCPToolset

Toolset that connects to an MCP server configured via an Airflow connection.

airflow.providers.common.ai.toolsets.mcp.MCPToolset

View Docs Source

SandboxToolset

Give an agent shell and file access inside a disposable sandbox, off the Airflow worker.

airflow.providers.common.ai.toolsets.sandbox.SandboxToolset

View Docs Source

AgentSkillsToolset

A pydantic-ai toolset that loads Agent Skills, with Git credentials from Airflow connections.

airflow.providers.common.ai.toolsets.skills.AgentSkillsToolset

View Docs Source

FailoverManagedAgentToolset

Present several interchangeable managed agents to the model as one tool.

airflow.providers.common.ai.toolsets.managed_agent.FailoverManagedAgentToolset

View Docs Source

ClassifierRetryPolicy

Retry policy where the model names the kind of failure and the author's table decides.

airflow.providers.common.ai.policies.retry.ClassifierRetryPolicy

View Docs Source

LLMRetryPolicy

Retry policy that uses an LLM to classify errors and decide retry behaviour.

airflow.providers.common.ai.policies.retry.LLMRetryPolicy

View Docs Source

HITLReviewPlugin

airflow.providers.common.ai.plugins.hitl_review.HITLReviewPlugin

View Docs Source

@task.agent Durable

Wrap a function that returns a prompt into an agentic LLM task.

airflow.providers.common.ai.decorators.agent.agent_task

View Docs Source

@task.llm

Wrap a function that returns a prompt into a general-purpose LLM task.

airflow.providers.common.ai.decorators.llm.llm_task

View Docs Source

@task.llm_batch Deferrable

Wrap a function that returns a batch's inputs into an ``@task.llm_batch`` task.

airflow.providers.common.ai.decorators.llm_batch.llm_batch_task

View Docs Source

@task.llm_file_analysis

Wrap a callable that returns a prompt into an LLM-backed file-analysis task.

airflow.providers.common.ai.decorators.llm_file_analysis.llm_file_analysis_task

View Docs Source

@task.llm_branch

Wrap a function that returns a prompt into an LLM-driven branching task.

airflow.providers.common.ai.decorators.llm_branch.llm_branch_task

View Docs Source

@task.llm_sql

Wrap a function that returns a natural language prompt into an LLM SQL task.

airflow.providers.common.ai.decorators.llm_sql.llm_sql_task

View Docs Source

@task.llm_schema_compare

Wrap a function that returns a prompt into an LLM schema comparison task.

airflow.providers.common.ai.decorators.llm_schema_compare.llm_schema_compare_task

View Docs Source