• Home
• Python API Reference
• airflow.models
• airflow.models.xcom_arg

airflow.models.xcom_arg

Module Contents

Classes

XComArg	Reference to an XCom value pushed from another operator.
PlainXComArg	Reference to one single XCom without any additional semantics.
MapXComArg	An XCom reference with map() call(s) applied.
ZipXComArg	An XCom reference with zip() applied.

Functions

serialize_xcom_arg(value)	DAG serialization interface.
deserialize_xcom_arg(data, dag)	DAG serialization interface.

Attributes

MapCallables

airflow.models.xcom_arg.MapCallables

class airflow.models.xcom_arg.XComArg

Bases: airflow.utils.mixins.ResolveMixin, airflow.models.taskmixin.DependencyMixin

Reference to an XCom value pushed from another operator.

The implementation supports:

xcomarg >> op
xcomarg << op
op >> xcomarg   # By BaseOperator code
op << xcomarg   # By BaseOperator code

Example: The moment you get a result from any operator (decorated or regular) you can

any_op = AnyOperator()
xcomarg = XComArg(any_op)
# or equivalently
xcomarg = any_op.output
my_op = MyOperator()
my_op >> xcomarg

This object can be used in legacy Operators via Jinja.

Example: You can make this result to be part of any generated string:

any_op = AnyOperator()
xcomarg = any_op.output
op1 = MyOperator(my_text_message=f"the value is {xcomarg}")
op2 = MyOperator(my_text_message=f"the value is {xcomarg['topic']}")

Parameters

• operator – Operator instance to which the XComArg references.

• key – Key used to pull the XCom value. Defaults to XCOM_RETURN_KEY, i.e. the referenced operator’s return value.

property roots: list[airflow.models.taskmixin.DAGNode]

Required by TaskMixin

property leaves: list[airflow.models.taskmixin.DAGNode]

Required by TaskMixin

static iter_xcom_references(arg)

Return XCom references in an arbitrary value.

Recursively traverse arg and look for XComArg instances in any collection objects, and instances with template_fields set.

static apply_upstream_relationship(op, arg)

Set dependency for XComArgs.

This looks for XComArg objects in arg “deeply” (looking inside collections objects and classes decorated with template_fields), and sets the relationship to op on any found.

set_upstream(task_or_task_list, edge_modifier=None)

Proxy to underlying operator set_upstream method. Required by TaskMixin.

set_downstream(task_or_task_list, edge_modifier=None)

Proxy to underlying operator set_downstream method. Required by TaskMixin.

map(f)

zip(*others, fillvalue=NOTSET)

abstract get_task_map_length(run_id, *, session)

Inspect length of pushed value for task-mapping.

This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.

None may be returned if the depended XCom has not been pushed.

abstract resolve(context, session=NEW_SESSION)

Pull XCom value.

This should only be called during op.execute() with an appropriate context (e.g. generated from TaskInstance.get_template_context()). Although the ResolveMixin parent mixin also has a resolve protocol, this adds the optional session argument that some of the subclasses need.

class airflow.models.xcom_arg.PlainXComArg(operator, key=XCOM_RETURN_KEY)

Bases: XComArg

Reference to one single XCom without any additional semantics.

This class should not be accessed directly, but only through XComArg. The class inheritance chain and __new__ is implemented in this slightly convoluted way because we want to

1. Allow the user to continue using XComArg directly for the simple semantics (see documentation of the base class for details).

2. Make isinstance(thing, XComArg) be able to detect all kinds of XCom references.

3. Not allow many properties of PlainXComArg (including __getitem__ and __str__) to exist on other kinds of XComArg implementations since they don’t make sense.

__eq__(other)

Return self==value.

__getitem__(item)

Implements xcomresult[‘some_result_key’]

__iter__()

Override iterable protocol to raise error explicitly.

The default __iter__ implementation in Python calls __getitem__ with 0, 1, 2, etc. until it hits an IndexError. This does not work well with our custom __getitem__ implementation, and results in poor DAG-writing experience since a misplaced * expansion would create an infinite loop consuming the entire DAG parser.

This override catches the error eagerly, so an incorrectly implemented DAG fails fast and avoids wasting resources on nonsensical iterating.

__repr__()

Return repr(self).

__str__()

Backward compatibility for old-style jinja used in Airflow Operators

Example: to use XComArg at BashOperator:

BashOperator(cmd=f"... { xcomarg } ...")

Returns

Return type

str

iter_references()

Find underlying XCom references this contains.

This is used by the DAG parser to recursively find task dependencies.

map(f)

zip(*others, fillvalue=NOTSET)

get_task_map_length(run_id, *, session)

Inspect length of pushed value for task-mapping.

This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.

None may be returned if the depended XCom has not been pushed.

resolve(context, session=NEW_SESSION)

Pull XCom value.

This should only be called during op.execute() with an appropriate context (e.g. generated from TaskInstance.get_template_context()). Although the ResolveMixin parent mixin also has a resolve protocol, this adds the optional session argument that some of the subclasses need.

class airflow.models.xcom_arg.MapXComArg(arg, callables)

Bases: XComArg

An XCom reference with map() call(s) applied.

This is based on an XComArg, but also applies a series of “transforms” that convert the pulled XCom value.

__repr__()

Return repr(self).

iter_references()

Find underlying XCom references this contains.

This is used by the DAG parser to recursively find task dependencies.

map(f)

get_task_map_length(run_id, *, session)

Inspect length of pushed value for task-mapping.

This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.

None may be returned if the depended XCom has not been pushed.

resolve(context, session=NEW_SESSION)

Pull XCom value.

This should only be called during op.execute() with an appropriate context (e.g. generated from TaskInstance.get_template_context()). Although the ResolveMixin parent mixin also has a resolve protocol, this adds the optional session argument that some of the subclasses need.

class airflow.models.xcom_arg.ZipXComArg(args, *, fillvalue=NOTSET)

Bases: XComArg

An XCom reference with zip() applied.

This is constructed from multiple XComArg instances, and presents an iterable that “zips” them together like the built-in zip() (and itertools.zip_longest() if fillvalue is provided).

__repr__()

Return repr(self).

iter_references()

Find underlying XCom references this contains.

This is used by the DAG parser to recursively find task dependencies.

get_task_map_length(run_id, *, session)

Inspect length of pushed value for task-mapping.

This is used to determine how many task instances the scheduler should create for a downstream using this XComArg for task-mapping.

None may be returned if the depended XCom has not been pushed.

resolve(context, session=NEW_SESSION)

Pull XCom value.

This should only be called during op.execute() with an appropriate context (e.g. generated from TaskInstance.get_template_context()). Although the ResolveMixin parent mixin also has a resolve protocol, this adds the optional session argument that some of the subclasses need.

airflow.models.xcom_arg.serialize_xcom_arg(value)

DAG serialization interface.

airflow.models.xcom_arg.deserialize_xcom_arg(data, dag)

DAG serialization interface.

Previous Next

Was this entry helpful?