airflow.providers.openai.operators.openai¶
Classes¶
Operator that accepts input text to generate OpenAI embeddings using the specified model. |
|
Operator that generates a model response using the OpenAI Responses API. |
|
Operator that triggers an OpenAI Batch API endpoint and waits for the batch to complete. |
Module Contents¶
- class airflow.providers.openai.operators.openai.OpenAIEmbeddingOperator(conn_id, input_text, model='text-embedding-3-small', embedding_kwargs=None, **kwargs)[source]¶
Bases:
airflow.providers.common.compat.sdk.BaseOperatorOperator that accepts input text to generate OpenAI embeddings using the specified model.
- Parameters:
conn_id (str) – The OpenAI connection ID to use.
input_text (str | list[str] | list[int] | list[list[int]]) – The text to generate OpenAI embeddings for. This can be a string, a list of strings, a list of integers, or a list of lists of integers.
model (str) – The OpenAI model to be used for generating the embeddings.
embedding_kwargs (dict | None) – Additional keyword arguments to pass to the OpenAI create_embeddings method.
Returns one embedding for a single string or token array, and one embedding per item for a batch.
See also
For more information on how to use this operator, take a look at the guide: OpenAIEmbeddingOperator For possible options for embedding_kwargs, see: https://platform.openai.com/docs/api-reference/embeddings/create
- template_fields: collections.abc.Sequence[str] = ('input_text', 'conn_id')[source]¶
- property hook: airflow.providers.openai.hooks.openai.OpenAIHook[source]¶
Return an instance of the OpenAIHook.
- class airflow.providers.openai.operators.openai.OpenAIResponseOperator(conn_id, input_text, model='gpt-4o-mini', response_kwargs=None, *, max_output_tokens=None, max_tool_calls=None, text_format=None, **kwargs)[source]¶
Bases:
airflow.providers.common.compat.sdk.BaseOperatorOperator that generates a model response using the OpenAI Responses API.
The operator is synchronous and returns the response’s aggregated output text, or, when
text_formatis set, the structured output parsed into that Pydantic model (seetext_formatbelow). The response id is also pushed to XCom (see below), so a downstream task can pick it up forprevious_response_idchaining without going through the hook. Forbackground=Trueresponses, or access to the full response object, useOpenAIHookdirectly.max_output_tokenscaps the number of tokens generated for the response;max_tool_callscaps the number of built-in tool calls the model may make. Both limits are enforced by the OpenAI API itself – OpenAI exposes no monetary cost limit on the Responses API, so this operator has no cost cap. For a monetary limit, use apache-airflow-providers-common-ai instead. Whenmax_output_tokensis hit, the request does not fail: the response comes back withstatus="incomplete"– butoutput_textis not guaranteed to contain any content, since a reasoning model can spend the entire ceiling on reasoning tokens without producing visible output. Hittingmax_tool_callsis different: the OpenAI SDK documents it as silently dropping further tool calls, with nostatuschange and noincomplete_details– a run truncated this way looks identical to a clean one in both the logs andreturn_value.- Parameters:
conn_id (str) – The OpenAI connection ID to use.
input_text (str | list[Any]) – The input prompt for the model. This can be a string or a structured list of input items.
model (str) – The OpenAI model to use.
response_kwargs (dict | None) – Additional keyword arguments to pass to the OpenAI
create_responsemethod, orparse_responsewhentext_formatis set (for exampleinstructions,tools,conversationorprevious_response_id). Templated, so values (e.g.previous_response_id) may reference upstream XCom. Do not setbackgroundorstreamhere:background=Truereturns before the response completes, so this operator logs a warning and the returned output text may be empty (withtext_formatset, the parsed response comes backqueuedorin_progress, so the task raisesValueErrorand the background response is left running on OpenAI’s side), whilestream=Truereturns an object withoutstatusoroutput_text, so the task raisesAttributeError. See OpenAIResponseOperator for these and other options this operator can pass through, such astruncationandmetadata.max_output_tokensandmax_tool_callsare special-cased when present as keys here – see their own:param:entries below for the exact validation, coercion, and blank-as-unset rules that apply to them.max_output_tokens (int | str | None) – Optional upper bound on the number of tokens generated for the response. Templated, so it renders to a string; accepts an
intor a string containing one. Must be a positive integer – an invalid value raises instead of silently disabling the ceiling. A literalbool,float, orintvalue is validated when the operator is constructed; any other non-string literal (for exampleDecimalorFraction) is coerced – and rejected if invalid – only when the task executes. A string value – whether a template or a plain literal string – is also validated when the task executes, after templating has resolved it. A blank or whitespace-only rendered value (for example{{ params.tokens | default('', true) }}rendering to'') is treated as unset, disabling the ceiling; the literal strings"None","none"and"null"are not treated as blank and still raise. A value that was supplied but resolves toNone(for example an unresolvedXComArg, or a Jinja-native-rendered null) also raises – it is not treated as unset. Mutually exclusive withmax_output_tokensinresponse_kwargs– this is checked when the operator is constructed, regardless of what the templated value later renders to. These same validation, type-coercion, blank-as-unset, and supplied-but-None-raises rules apply identically whenmax_output_tokensis set as a key directly insideresponse_kwargsinstead of passed as this operator argument – with one difference: for that key, blank-as-unset means the key is removed from the payload passed tocreate_response(rather than never being added, as happens for this operator argument), and a present key whose value is a literalNonestill raises, since presence of the key – not the value – is what “supplied” means for that path.max_tool_calls (int | str | None) – Optional upper bound on the number of built-in tool calls the model may make while generating the response. Same templating, type, validation, blank-as-unset, and mutual-exclusion rules as
max_output_tokens.text_format (type[pydantic.BaseModel] | None) – Optional Pydantic
BaseModelsubclass describing the expected structured output. When set, the operator callsparse_responseinstead ofcreate_responseand returns the parsed model’smodel_dump(mode="json"), so enums, dates and other non-JSON field types reach XCom as their JSON representations. The task fails withValueErrorrather than returning partial data when the response did not complete (for example becausemax_output_tokenswas reached), when it carries no parsed output (for example a refusal), or when the SDK cannot validate the output against the model. Withmultiple_outputs=Trueeach top-level field is pushed as its own XCom afterexecutereturns, soresponse_idandusageare reserved field names: a field with either name would overwrite the XCom this operator pushes under that key.
See also
For more information on how to use this operator, take a look at the guide: OpenAIResponseOperator For possible options, see: https://platform.openai.com/docs/api-reference/responses/create
When
do_xcom_pushis enabled (the default),executealso pushes two XCom keys:response_id(the response’s ID) andusage(the result ofResponseUsage.model_dump(), orNonewhen the API omits it). Whenusageis notNoneit also carries atry_numberkey recording which attempt produced it – XCom is cleared at the start of every attempt, so this makes it visible that the value only reflects the current attempt rather than a silently under-reported total across retries. Withtext_formatset, both keys are pushed before the structured output is checked, so a response that then fails the task still records its id and token usage. The exception is output the SDK cannot parse, such as JSON cut off mid-object: it raises before returning a response object, so there is no id or usage to push and neither key is recorded. Both XCom pushes are skipped whendo_xcom_push=False.- template_fields: collections.abc.Sequence[str] = ('input_text', 'response_kwargs', 'max_output_tokens', 'max_tool_calls', 'conn_id')[source]¶
- property hook: airflow.providers.openai.hooks.openai.OpenAIHook[source]¶
Return an instance of the OpenAIHook.
- class airflow.providers.openai.operators.openai.OpenAITriggerBatchOperator(file_id, endpoint, conn_id=OpenAIHook.default_conn_name, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), wait_seconds=3, timeout=24 * 60 * 60, wait_for_completion=True, *, metadata=None, batch_kwargs=None, poll_interval=60, **kwargs)[source]¶
Bases:
airflow.providers.common.compat.sdk.BaseOperatorOperator that triggers an OpenAI Batch API endpoint and waits for the batch to complete.
- Parameters:
file_id (str) – Required. The ID of the batch file to trigger. (templated)
endpoint (str) – Required. The OpenAI Batch API endpoint to trigger. (templated) Allowed values are determined by the OpenAI Batch API; see
create_batch().conn_id (str) – Optional. The OpenAI connection ID to use. Defaults to ‘openai_default’.
deferrable (bool) – Optional. Run operator in the deferrable mode.
wait_seconds (float) – Optional. Number of seconds between checks. Only used when
deferrableis False. Defaults to 3 seconds.timeout (float) – Optional. The number of seconds to wait for the batch to complete, in both deferrable and non-deferrable mode. Defaults to 24 hours, the SLA for OpenAI Batch API. In deferrable mode, if
execution_timeoutis set shorter thantimeout, the task is failed withTaskDeferralTimeoutbefore the trigger times out, and the batch is not cancelled.wait_for_completion (bool) – Optional. Whether to wait for the batch to complete. If set to False, the operator will return immediately after triggering the batch. Defaults to True.
metadata (dict[str, str] | None) – Optional. A set of key-value pairs that can be attached to the batch. (templated)
batch_kwargs (dict | None) – Optional. Additional keyword arguments to pass to the OpenAI create_batch method — for example output_expires_after, which sets the expiry on the batch’s output and error files. Defaults to None.
poll_interval (float) – Optional. Number of seconds between checks. Only used when
deferrableis True. Defaults to 60 seconds.
See also
For more information on how to use this operator, please take a look at the guide: OpenAITriggerBatchOperator
- template_fields: collections.abc.Sequence[str] = ('file_id', 'endpoint', 'metadata', 'conn_id')[source]¶
- property hook: airflow.providers.openai.hooks.openai.OpenAIHook[source]¶
Return an instance of the OpenAIHook.
- execute(context)[source]¶
Derive when creating an operator.
The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.
Refer to get_template_context for more context.
- execute_complete(context, event=None)[source]¶
Invoke this callback when the trigger fires; return immediately.
Relies on trigger to throw an exception, otherwise it assumes execution was successful. The exception raised depends on the event’s
termination_reason:OpenAIBatchTimeoutfor a timeout (matching the exception the synchronous path raises for the same condition),OpenAIBatchCancelledfor a cancellation (a subclass ofOpenAIBatchJobException), andOpenAIBatchJobExceptionfor any other failure (including events from a trigger serialized beforetermination_reasonexisted).On a timeout, cancellation of the batch is requested before the timeout is raised (see
_cancel_batch_quietly()). No other termination reason triggers cancellation: apolling_errormay be a transient, Airflow-side failure rather than a real batch problem, and cancellation is irreversible, so it is left alone to run to its own 24-hour completion window instead.