airflow.providers.openai.operators.openai

Classes

OpenAIEmbeddingOperator

Operator that accepts input text to generate OpenAI embeddings using the specified model.

OpenAIResponseOperator

Operator that generates a model response using the OpenAI Responses API.

OpenAITriggerBatchOperator

Operator that triggers an OpenAI Batch API endpoint and waits for the batch to complete.

Module Contents

class airflow.providers.openai.operators.openai.OpenAIEmbeddingOperator(conn_id, input_text, model='text-embedding-3-small', embedding_kwargs=None, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Operator that accepts input text to generate OpenAI embeddings using the specified model.

Parameters:
  • conn_id (str) – The OpenAI connection ID to use.

  • input_text (str | list[str] | list[int] | list[list[int]]) – The text to generate OpenAI embeddings for. This can be a string, a list of strings, a list of integers, or a list of lists of integers.

  • model (str) – The OpenAI model to be used for generating the embeddings.

  • embedding_kwargs (dict | None) – Additional keyword arguments to pass to the OpenAI create_embeddings method.

Returns one embedding for a single string or token array, and one embedding per item for a batch.

See also

For more information on how to use this operator, take a look at the guide: OpenAIEmbeddingOperator For possible options for embedding_kwargs, see: https://platform.openai.com/docs/api-reference/embeddings/create

template_fields: collections.abc.Sequence[str] = ('input_text', 'conn_id')[source]
conn_id[source]
input_text[source]
model = 'text-embedding-3-small'[source]
embedding_kwargs[source]
property hook: airflow.providers.openai.hooks.openai.OpenAIHook[source]

Return an instance of the OpenAIHook.

execute(context)[source]

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.openai.operators.openai.OpenAIResponseOperator(conn_id, input_text, model='gpt-4o-mini', response_kwargs=None, *, max_output_tokens=None, max_tool_calls=None, text_format=None, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Operator that generates a model response using the OpenAI Responses API.

The operator is synchronous and returns the response’s aggregated output text, or, when text_format is set, the structured output parsed into that Pydantic model (see text_format below). The response id is also pushed to XCom (see below), so a downstream task can pick it up for previous_response_id chaining without going through the hook. For background=True responses, or access to the full response object, use OpenAIHook directly.

max_output_tokens caps the number of tokens generated for the response; max_tool_calls caps the number of built-in tool calls the model may make. Both limits are enforced by the OpenAI API itself – OpenAI exposes no monetary cost limit on the Responses API, so this operator has no cost cap. For a monetary limit, use apache-airflow-providers-common-ai instead. When max_output_tokens is hit, the request does not fail: the response comes back with status="incomplete" – but output_text is not guaranteed to contain any content, since a reasoning model can spend the entire ceiling on reasoning tokens without producing visible output. Hitting max_tool_calls is different: the OpenAI SDK documents it as silently dropping further tool calls, with no status change and no incomplete_details – a run truncated this way looks identical to a clean one in both the logs and return_value.

Parameters:
  • conn_id (str) – The OpenAI connection ID to use.

  • input_text (str | list[Any]) – The input prompt for the model. This can be a string or a structured list of input items.

  • model (str) – The OpenAI model to use.

  • response_kwargs (dict | None) – Additional keyword arguments to pass to the OpenAI create_response method, or parse_response when text_format is set (for example instructions, tools, conversation or previous_response_id). Templated, so values (e.g. previous_response_id) may reference upstream XCom. Do not set background or stream here: background=True returns before the response completes, so this operator logs a warning and the returned output text may be empty (with text_format set, the parsed response comes back queued or in_progress, so the task raises ValueError and the background response is left running on OpenAI’s side), while stream=True returns an object without status or output_text, so the task raises AttributeError. See OpenAIResponseOperator for these and other options this operator can pass through, such as truncation and metadata. max_output_tokens and max_tool_calls are special-cased when present as keys here – see their own :param: entries below for the exact validation, coercion, and blank-as-unset rules that apply to them.

  • max_output_tokens (int | str | None) – Optional upper bound on the number of tokens generated for the response. Templated, so it renders to a string; accepts an int or a string containing one. Must be a positive integer – an invalid value raises instead of silently disabling the ceiling. A literal bool, float, or int value is validated when the operator is constructed; any other non-string literal (for example Decimal or Fraction) is coerced – and rejected if invalid – only when the task executes. A string value – whether a template or a plain literal string – is also validated when the task executes, after templating has resolved it. A blank or whitespace-only rendered value (for example {{ params.tokens | default('', true) }} rendering to '') is treated as unset, disabling the ceiling; the literal strings "None", "none" and "null" are not treated as blank and still raise. A value that was supplied but resolves to None (for example an unresolved XComArg, or a Jinja-native-rendered null) also raises – it is not treated as unset. Mutually exclusive with max_output_tokens in response_kwargs – this is checked when the operator is constructed, regardless of what the templated value later renders to. These same validation, type-coercion, blank-as-unset, and supplied-but-None-raises rules apply identically when max_output_tokens is set as a key directly inside response_kwargs instead of passed as this operator argument – with one difference: for that key, blank-as-unset means the key is removed from the payload passed to create_response (rather than never being added, as happens for this operator argument), and a present key whose value is a literal None still raises, since presence of the key – not the value – is what “supplied” means for that path.

  • max_tool_calls (int | str | None) – Optional upper bound on the number of built-in tool calls the model may make while generating the response. Same templating, type, validation, blank-as-unset, and mutual-exclusion rules as max_output_tokens.

  • text_format (type[pydantic.BaseModel] | None) – Optional Pydantic BaseModel subclass describing the expected structured output. When set, the operator calls parse_response instead of create_response and returns the parsed model’s model_dump(mode="json"), so enums, dates and other non-JSON field types reach XCom as their JSON representations. The task fails with ValueError rather than returning partial data when the response did not complete (for example because max_output_tokens was reached), when it carries no parsed output (for example a refusal), or when the SDK cannot validate the output against the model. With multiple_outputs=True each top-level field is pushed as its own XCom after execute returns, so response_id and usage are reserved field names: a field with either name would overwrite the XCom this operator pushes under that key.

See also

For more information on how to use this operator, take a look at the guide: OpenAIResponseOperator For possible options, see: https://platform.openai.com/docs/api-reference/responses/create

When do_xcom_push is enabled (the default), execute also pushes two XCom keys: response_id (the response’s ID) and usage (the result of ResponseUsage.model_dump(), or None when the API omits it). When usage is not None it also carries a try_number key recording which attempt produced it – XCom is cleared at the start of every attempt, so this makes it visible that the value only reflects the current attempt rather than a silently under-reported total across retries. With text_format set, both keys are pushed before the structured output is checked, so a response that then fails the task still records its id and token usage. The exception is output the SDK cannot parse, such as JSON cut off mid-object: it raises before returning a response object, so there is no id or usage to push and neither key is recorded. Both XCom pushes are skipped when do_xcom_push=False.

template_fields: collections.abc.Sequence[str] = ('input_text', 'response_kwargs', 'max_output_tokens', 'max_tool_calls', 'conn_id')[source]
conn_id[source]
input_text[source]
model = 'gpt-4o-mini'[source]
response_kwargs[source]
max_output_tokens = None[source]
max_tool_calls = None[source]
text_format = None[source]
property hook: airflow.providers.openai.hooks.openai.OpenAIHook[source]

Return an instance of the OpenAIHook.

execute(context)[source]

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

class airflow.providers.openai.operators.openai.OpenAITriggerBatchOperator(file_id, endpoint, conn_id=OpenAIHook.default_conn_name, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), wait_seconds=3, timeout=24 * 60 * 60, wait_for_completion=True, *, metadata=None, batch_kwargs=None, poll_interval=60, **kwargs)[source]

Bases: airflow.providers.common.compat.sdk.BaseOperator

Operator that triggers an OpenAI Batch API endpoint and waits for the batch to complete.

Parameters:
  • file_id (str) – Required. The ID of the batch file to trigger. (templated)

  • endpoint (str) – Required. The OpenAI Batch API endpoint to trigger. (templated) Allowed values are determined by the OpenAI Batch API; see create_batch().

  • conn_id (str) – Optional. The OpenAI connection ID to use. Defaults to ‘openai_default’.

  • deferrable (bool) – Optional. Run operator in the deferrable mode.

  • wait_seconds (float) – Optional. Number of seconds between checks. Only used when deferrable is False. Defaults to 3 seconds.

  • timeout (float) – Optional. The number of seconds to wait for the batch to complete, in both deferrable and non-deferrable mode. Defaults to 24 hours, the SLA for OpenAI Batch API. In deferrable mode, if execution_timeout is set shorter than timeout, the task is failed with TaskDeferralTimeout before the trigger times out, and the batch is not cancelled.

  • wait_for_completion (bool) – Optional. Whether to wait for the batch to complete. If set to False, the operator will return immediately after triggering the batch. Defaults to True.

  • metadata (dict[str, str] | None) – Optional. A set of key-value pairs that can be attached to the batch. (templated)

  • batch_kwargs (dict | None) – Optional. Additional keyword arguments to pass to the OpenAI create_batch method — for example output_expires_after, which sets the expiry on the batch’s output and error files. Defaults to None.

  • poll_interval (float) – Optional. Number of seconds between checks. Only used when deferrable is True. Defaults to 60 seconds.

See also

For more information on how to use this operator, please take a look at the guide: OpenAITriggerBatchOperator

template_fields: collections.abc.Sequence[str] = ('file_id', 'endpoint', 'metadata', 'conn_id')[source]
template_fields_renderers[source]
file_id[source]
endpoint[source]
conn_id = 'openai_default'[source]
deferrable[source]
wait_seconds = 3[source]
timeout = 86400[source]
wait_for_completion = True[source]
metadata = None[source]
batch_kwargs[source]
poll_interval = 60[source]
batch_id: str | None = None[source]
property hook: airflow.providers.openai.hooks.openai.OpenAIHook[source]

Return an instance of the OpenAIHook.

execute(context)[source]

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]

Invoke this callback when the trigger fires; return immediately.

Relies on trigger to throw an exception, otherwise it assumes execution was successful. The exception raised depends on the event’s termination_reason: OpenAIBatchTimeout for a timeout (matching the exception the synchronous path raises for the same condition), OpenAIBatchCancelled for a cancellation (a subclass of OpenAIBatchJobException), and OpenAIBatchJobException for any other failure (including events from a trigger serialized before termination_reason existed).

On a timeout, cancellation of the batch is requested before the timeout is raised (see _cancel_batch_quietly()). No other termination reason triggers cancellation: a polling_error may be a transient, Airflow-side failure rather than a real batch problem, and cancellation is irreversible, so it is left alone to run to its own 24-hour completion window instead.

on_kill()[source]

Cancel the batch if task is cancelled.

Was this entry helpful?