airflow.models.mappedoperator

Module Contents

Classes

OperatorPartial

An "intermediate state" returned by BaseOperator.partial().

MappedOperator

Object representing a mapped operator in a DAG.

Functions

get_mappable_types()

validate_mapping_kwargs(op, func, value)

prevent_duplicates(kwargs1, kwargs2, *, fail_reason)

ensure_xcomarg_return_value(arg)

Attributes

Mappable

ValidationSource

MAPPABLE_LITERAL_TYPES

airflow.models.mappedoperator.Mappable[source]
airflow.models.mappedoperator.ValidationSource[source]
airflow.models.mappedoperator.MAPPABLE_LITERAL_TYPES[source]
airflow.models.mappedoperator.get_mappable_types()[source]
airflow.models.mappedoperator.validate_mapping_kwargs(op, func, value)[source]
airflow.models.mappedoperator.prevent_duplicates(kwargs1, kwargs2, *, fail_reason)[source]
airflow.models.mappedoperator.ensure_xcomarg_return_value(arg)[source]
class airflow.models.mappedoperator.OperatorPartial[source]

An "intermediate state" returned by BaseOperator.partial().

This only exists at DAG-parsing time; the only intended usage is for the user to call .expand() on it at some point (usually in a method chain) to create a MappedOperator to add into the DAG.

operator_class :Type[airflow.models.baseoperator.BaseOperator][source]
kwargs :Dict[str, Any][source]
__attrs_post_init__(self)[source]
__repr__(self)[source]

Return repr(self).

__del__(self)[source]
expand(self, **mapped_kwargs)[source]
class airflow.models.mappedoperator.MappedOperator(context=None)[source]

Bases: airflow.models.abstractoperator.AbstractOperator

Object representing a mapped operator in a DAG.

operator_class :Union[Type[airflow.models.baseoperator.BaseOperator], Dict[str, Any]][source]
mapped_kwargs :Dict[str, Mappable][source]
partial_kwargs :Dict[str, Any][source]
task_id :str[source]
params :Optional[dict][source]
deps :FrozenSet[airflow.ti_deps.deps.base_ti_dep.BaseTIDep][source]
template_ext :Sequence[str][source]
template_fields :Collection[str][source]
template_fields_renderers :Dict[str, str][source]
ui_color :str[source]
ui_fgcolor :str[source]
dag :Optional[airflow.models.dag.DAG][source]
task_group :Optional[airflow.utils.task_group.TaskGroup][source]
start_date :Optional[pendulum.DateTime][source]
end_date :Optional[pendulum.DateTime][source]
upstream_task_ids :Set[str][source]
downstream_task_ids :Set[str][source]
is_mapped :ClassVar[bool] = True[source]
subdag :None[source]
HIDE_ATTRS_FROM_UI :ClassVar[FrozenSet[str]][source]
__hash__(self)[source]

Return hash(self).

__repr__(self)[source]

Return repr(self).

__attrs_post_init__(self)[source]
classmethod get_serialized_fields(cls)[source]
static deps_for(operator_class)[source]
property task_type(self)[source]

Implementing Operator.

property inherits_from_empty_operator(self)[source]

Implementing Operator.

property roots(self)[source]

Implementing DAGNode.

property leaves(self)[source]

Implementing DAGNode.

property owner(self)[source]
property email(self)[source]
property trigger_rule(self)[source]
property depends_on_past(self)[source]
property wait_for_downstream(self)[source]
property retries(self)[source]
property queue(self)[source]
property pool(self)[source]
property pool_slots(self)[source]
property execution_timeout(self)[source]
property retry_delay(self)[source]
property retry_exponential_backoff(self)[source]
property priority_weight(self)[source]
property weight_rule(self)[source]
property sla(self)[source]
property max_active_tis_per_dag(self)[source]
property resources(self)[source]
property on_execute_callback(self)[source]
property on_failure_callback(self)[source]
property on_retry_callback(self)[source]
property on_success_callback(self)[source]
property run_as_user(self)[source]
property executor_config(self)[source]
property inlets(self)[source]
property outlets(self)[source]
get_dag(self)[source]

Implementing Operator.

serialize_for_task_group(self)[source]

Implementing DAGNode.

unmap(self, unmap_kwargs=None)[source]

Get the "normal" Operator after applying the current mapping.

If operator_class is not a class (i.e. this DAG has been deserialized) then this will return a SerializedBaseOperator that aims to "look like" the real operator.

Parameters

unmap_kwargs (Optional[Dict[str, Any]]) -- Override the args to pass to the Operator constructor. Only used when operator_class is still an actual class.

expand_mapped_task(self, run_id, *, session)[source]

Create the mapped task instances for mapped task.

Returns

The newly created mapped TaskInstances (if any) in ascending order by map index, and the maximum map_index.

Return type

Tuple[Sequence[airflow.models.taskinstance.TaskInstance], int]

prepare_for_execution(self)[source]
render_template_fields(self, context, jinja_env=None)[source]

Template all attributes listed in template_fields.

Different from the BaseOperator implementation, this renders the template fields on the unmapped BaseOperator.

Parameters

  • context (airflow.utils.context.Context) -- Dict with values to apply on content

  • jinja_env (Optional[jinja2.Environment]) -- Jinja environment

Returns

The unmapped, populated BaseOperator

Return type

Optional[airflow.models.baseoperator.BaseOperator]

iter_mapped_dependencies(self)[source]

Upstream dependencies that provide XComs used by this task for task mapping.

parse_time_mapped_ti_count(self)[source]

Number of mapped TaskInstances that can be created at DagRun create time.

Returns

None if non-literal mapped arg encountered, or else total number of mapped TIs this task should have

Return type

Optional[int]

run_time_mapped_ti_count(self, run_id, *, session)[source]

Number of mapped TaskInstances that can be created at run time, or None if upstream tasks are not complete yet.

Returns

None if upstream tasks are not complete yet, or else total number of mapped TIs this task should have

Return type

Optional[int]

Was this entry helpful?