Dag
Source: dag.ts:679
A Dag declared in TypeScript.
Declare the tasks with dag.task(taskId, handler), then call what each returns to lay the Dag out:
const extract = dag.task("extract", async (): Promise<number> => 42);
const load = dag.task("load", async ({ rows }: { rows: number }) => {});
load({ rows: extract() });Every task has to be called exactly once, so none can be left out of the Dag by accident. A TypeScript handler for a task that a Python Dag declares is a TaskHandler instead, not a task on a Dag.
Constructing a Dag has no effect beyond the instance itself. Collect the ones a bundle should serve on a Bundle and await bundle.serve().
Constructors
new Dag(dagId: string, spec: DagSpec): Dag
Parameters
dagId(string)spec(DagSpec, default: "{}")
Returns
Dag
Properties
dagId: string
Identifier of this Dag.
spec: DagSpec
Dag-level options this instance was constructed with, copied and frozen.
Accessors
get taskIds(): readonly string[]
Task IDs attached to this Dag, in attachment order.
Methods
if<
TArgs extends void | object = void,
>(
handler: (args: TArgs) => boolean | Promise<boolean>,
...args: DeciderArgs<NoInfer<TArgs>>,
): Condition
Declare a task whose boolean picks a branch; the side not taken is skipped.
dag.if(hasRows, { rows: validated }).then(loadIfReady).else(loadFallback);
dag.if(hasRows, { rows: validated }, { taskId: "has_rows", retries: 2 }).then(loadIfReady);Type Parameters
TArgsextendsvoid | object=void
Parameters
handler((args: TArgs) => boolean | Promise<boolean>)args(DeciderArgs<NoInfer<TArgs>>)
Returns
Condition
switch<
TArgs extends void | object = void,
>(
handler: (args: TArgs) => TaskRef<unknown> | Promise<TaskRef<unknown>>,
...args: DeciderArgs<NoInfer<TArgs>>,
): Branch
Declare a task that returns one of its cases; every other case is skipped.
dag.switch(pickPath, { rows: extracted }).case(handleLong).case(handleShort);Type Parameters
TArgsextendsvoid | object=void
Parameters
handler((args: TArgs) => TaskRef<unknown> | Promise<TaskRef<unknown>>)args(DeciderArgs<NoInfer<TArgs>>)
Returns
Branch
task
Declare a task of this Dag, and return the factory that places it.
A handler takes one object of named arguments, and every argument in it becomes an input of the returned TaskFactory; getContext() and getClient() reach the runtime from inside the call, so neither is an argument. The trailing options object carries this task's own TaskSpec.
const extract = dag.task("extract", async () => 42);
const transform = dag.task(async function transform() {}); // id "transform"task<
TArgs extends void | object = void,
TReturn = unknown,
>(
taskId: string,
handler: (args: TArgs) => TReturn | Promise<TReturn>,
options?: TaskSpec,
): TaskFactory<TArgs, TReturn>Type Parameters
TArgsextendsvoid | object=voidTReturn=unknown
Parameters
taskId(string)handler((args: TArgs) => TReturn | Promise<TReturn>)options(TaskSpec, optional)
Returns
TaskFactory<TArgs, TReturn>
task<
TArgs extends void | object = void,
TReturn = unknown,
>(
handler: (args: TArgs) => TReturn | Promise<TReturn>,
options?: TaskSpec,
): TaskFactory<TArgs, TReturn>Declare a task whose id is the handler's function name.
airflow-ts-pack keeps handler names intact, so minification cannot change a task id. An anonymous handler has no name to take one from, and needs TaskSpec.taskId to give it one.
Type Parameters
TArgsextendsvoid | object=voidTReturn=unknown
Parameters
handler((args: TArgs) => TReturn | Promise<TReturn>)options(TaskSpec, optional)
Returns
TaskFactory<TArgs, TReturn>
task<
TArgs extends void | object = void,
TResult = unknown,
>(
taskId: string,
operator: Operator<TArgs, TResult>,
options?: TaskSpec,
): TaskFactory<TArgs, TResult>Declare a task an operator runs, such as triggerDagRun(spec). Its inputs and result are the operator's own. It has no handler to take an id from, so the id is required.
Type Parameters
TArgsextendsvoid | object=voidTResult=unknown
Parameters
taskId(string)operator(Operator<TArgs, TResult>)options(TaskSpec, optional)
Returns
TaskFactory<TArgs, TResult>
task<
TArgs extends void | object = void,
TResult = unknown,
>(
operator: Operator<TArgs, TResult>,
options: TaskSpec,
): TaskFactory<TArgs, TResult>Declare a task of this Dag, and return the factory that places it.
A handler takes one object of named arguments, and every argument in it becomes an input of the returned TaskFactory; getContext() and getClient() reach the runtime from inside the call, so neither is an argument. The trailing options object carries this task's own TaskSpec.
const extract = dag.task("extract", async () => 42);
const transform = dag.task(async function transform() {}); // id "transform"Type Parameters
TArgsextendsvoid | object=voidTResult=unknown
Parameters
operator(Operator<TArgs, TResult>)options(TaskSpec)
Returns
TaskFactory<TArgs, TResult>
taskGroup(
groupId: string,
options?: TaskGroupOptions,
): TaskGroupRef
Declare a task group of this Dag.
The group prefixes the id of everything declared in it, unless TaskGroupOptions.prefixGroupId turns that off, and stands at either end of an order-only edge in its own right.
const staging = dag.taskGroup("staging"); // tasks "staging.<id>"
const checks = dag.taskGroup("checks", { prefixGroupId: false }); // ids as writtenParameters
groupId(string)options(TaskGroupOptions, optional)
Returns
TaskGroupRef