airflow.models.xcom_arg
¶
Module Contents¶
Classes¶
Reference to an XCom value pushed from another operator. |
|
Reference to one single XCom without any additional semantics. |
|
An XCom reference with |
|
An XCom reference with |
Functions¶
|
DAG serialization interface. |
|
DAG serialization interface. |
Attributes¶
- class airflow.models.xcom_arg.XComArg[source]¶
Bases:
airflow.utils.mixins.ResolveMixin
,airflow.models.taskmixin.DependencyMixin
Reference to an XCom value pushed from another operator.
The implementation supports:
xcomarg >> op xcomarg << op op >> xcomarg # By BaseOperator code op << xcomarg # By BaseOperator code
Example: The moment you get a result from any operator (decorated or regular) you can
any_op = AnyOperator() xcomarg = XComArg(any_op) # or equivalently xcomarg = any_op.output my_op = MyOperator() my_op >> xcomarg
This object can be used in legacy Operators via Jinja.
Example: You can make this result to be part of any generated string:
any_op = AnyOperator() xcomarg = any_op.output op1 = MyOperator(my_text_message=f"the value is {xcomarg}") op2 = MyOperator(my_text_message=f"the value is {xcomarg['topic']}")
- Parameters
operator -- Operator instance to which the XComArg references.
key -- Key used to pull the XCom value. Defaults to XCOM_RETURN_KEY, i.e. the referenced operator's return value.
- property roots: list[airflow.models.taskmixin.DAGNode][source]¶
Required by TaskMixin
- property leaves: list[airflow.models.taskmixin.DAGNode][source]¶
Required by TaskMixin
- static iter_xcom_references(arg)[source]¶
Return XCom references in an arbitrary value.
Recursively traverse
arg
and look for XComArg instances in any collection objects, and instances withtemplate_fields
set.
- static apply_upstream_relationship(op, arg)[source]¶
Set dependency for XComArgs.
This looks for XComArg objects in
arg
"deeply" (looking inside collections objects and classes decorated withtemplate_fields
), and sets the relationship toop
on any found.
- set_upstream(task_or_task_list, edge_modifier=None)[source]¶
Proxy to underlying operator set_upstream method. Required by TaskMixin.
- set_downstream(task_or_task_list, edge_modifier=None)[source]¶
Proxy to underlying operator set_downstream method. Required by TaskMixin.
- abstract get_task_map_length(run_id, *, session)[source]¶
Inspect length of pushed value for task-mapping.
This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.
None may be returned if the depended XCom has not been pushed.
- abstract resolve(context, session=NEW_SESSION)[source]¶
Pull XCom value.
This should only be called during
op.execute()
with an appropriate context (e.g. generated fromTaskInstance.get_template_context()
). Although theResolveMixin
parent mixin also has aresolve
protocol, this adds the optionalsession
argument that some of the subclasses need.
- class airflow.models.xcom_arg.PlainXComArg(operator, key=XCOM_RETURN_KEY)[source]¶
Bases:
XComArg
Reference to one single XCom without any additional semantics.
This class should not be accessed directly, but only through XComArg. The class inheritance chain and
__new__
is implemented in this slightly convoluted way because we want toAllow the user to continue using XComArg directly for the simple semantics (see documentation of the base class for details).
Make
isinstance(thing, XComArg)
be able to detect all kinds of XCom references.Not allow many properties of PlainXComArg (including
__getitem__
and__str__
) to exist on other kinds of XComArg implementations since they don't make sense.
- __iter__()[source]¶
Override iterable protocol to raise error explicitly.
The default
__iter__
implementation in Python calls__getitem__
with 0, 1, 2, etc. until it hits anIndexError
. This does not work well with our custom__getitem__
implementation, and results in poor DAG-writing experience since a misplaced*
expansion would create an infinite loop consuming the entire DAG parser.This override catches the error eagerly, so an incorrectly implemented DAG fails fast and avoids wasting resources on nonsensical iterating.
- __str__()[source]¶
Backward compatibility for old-style jinja used in Airflow Operators
Example: to use XComArg at BashOperator:
BashOperator(cmd=f"... { xcomarg } ...")
- Returns
- Return type
- iter_references()[source]¶
Find underlying XCom references this contains.
This is used by the DAG parser to recursively find task dependencies.
- get_task_map_length(run_id, *, session)[source]¶
Inspect length of pushed value for task-mapping.
This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.
None may be returned if the depended XCom has not been pushed.
- resolve(context, session=NEW_SESSION)[source]¶
Pull XCom value.
This should only be called during
op.execute()
with an appropriate context (e.g. generated fromTaskInstance.get_template_context()
). Although theResolveMixin
parent mixin also has aresolve
protocol, this adds the optionalsession
argument that some of the subclasses need.
- class airflow.models.xcom_arg.MapXComArg(arg, callables)[source]¶
Bases:
XComArg
An XCom reference with
map()
call(s) applied.This is based on an XComArg, but also applies a series of "transforms" that convert the pulled XCom value.
- iter_references()[source]¶
Find underlying XCom references this contains.
This is used by the DAG parser to recursively find task dependencies.
- get_task_map_length(run_id, *, session)[source]¶
Inspect length of pushed value for task-mapping.
This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.
None may be returned if the depended XCom has not been pushed.
- resolve(context, session=NEW_SESSION)[source]¶
Pull XCom value.
This should only be called during
op.execute()
with an appropriate context (e.g. generated fromTaskInstance.get_template_context()
). Although theResolveMixin
parent mixin also has aresolve
protocol, this adds the optionalsession
argument that some of the subclasses need.
- class airflow.models.xcom_arg.ZipXComArg(args, *, fillvalue=NOTSET)[source]¶
Bases:
XComArg
An XCom reference with
zip()
applied.This is constructed from multiple XComArg instances, and presents an iterable that "zips" them together like the built-in
zip()
(anditertools.zip_longest()
iffillvalue
is provided).- iter_references()[source]¶
Find underlying XCom references this contains.
This is used by the DAG parser to recursively find task dependencies.
- get_task_map_length(run_id, *, session)[source]¶
Inspect length of pushed value for task-mapping.
This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.
None may be returned if the depended XCom has not been pushed.
- resolve(context, session=NEW_SESSION)[source]¶
Pull XCom value.
This should only be called during
op.execute()
with an appropriate context (e.g. generated fromTaskInstance.get_template_context()
). Although theResolveMixin
parent mixin also has aresolve
protocol, this adds the optionalsession
argument that some of the subclasses need.