Dag

Source: dag.ts:679

A Dag declared in TypeScript.

Declare the tasks with dag.task(taskId, handler), then call what each returns to lay the Dag out:

CODE
const extract = dag.task("extract", async (): Promise<number> => 42);
const load = dag.task("load", async ({ rows }: { rows: number }) => {});
load({ rows: extract() });

Every task has to be called exactly once, so none can be left out of the Dag by accident. A TypeScript handler for a task that a Python Dag declares is a TaskHandler instead, not a task on a Dag.

Constructing a Dag has no effect beyond the instance itself. Collect the ones a bundle should serve on a Bundle and await bundle.serve().


Constructors

new Dag(dagId: string, spec: DagSpec): Dag

Parameters

  • dagId (string)
  • spec (DagSpec, default: "{}")

Returns

Dag


Properties

dagId: string

readonly
dag.ts:681

Identifier of this Dag.

spec: DagSpec

readonly
dag.ts:683

Dag-level options this instance was constructed with, copied and frozen.

Accessors

get taskIds(): readonly string[]

Task IDs attached to this Dag, in attachment order.

Methods

if< TArgs extends void | object = void, >( handler: (args: TArgs) => boolean | Promise<boolean>, ...args: DeciderArgs<NoInfer<TArgs>>, ): Condition

Declare a task whose boolean picks a branch; the side not taken is skipped.

CODE
dag.if(hasRows, { rows: validated }).then(loadIfReady).else(loadFallback);
dag.if(hasRows, { rows: validated }, { taskId: "has_rows", retries: 2 }).then(loadIfReady);

Type Parameters

  • TArgs extends void | object = void

Parameters

  • handler ((args: TArgs) => boolean | Promise<boolean>)
  • args (DeciderArgs<NoInfer<TArgs>>)

Returns

  • Condition

switch< TArgs extends void | object = void, >( handler: (args: TArgs) => TaskRef<unknown> | Promise<TaskRef<unknown>>, ...args: DeciderArgs<NoInfer<TArgs>>, ): Branch

Declare a task that returns one of its cases; every other case is skipped.

CODE
dag.switch(pickPath, { rows: extracted }).case(handleLong).case(handleShort);

Type Parameters

  • TArgs extends void | object = void

Parameters

  • handler ((args: TArgs) => TaskRef<unknown> | Promise<TaskRef<unknown>>)
  • args (DeciderArgs<NoInfer<TArgs>>)

Returns

  • Branch

task

Declare a task of this Dag, and return the factory that places it.

A handler takes one object of named arguments, and every argument in it becomes an input of the returned TaskFactory; getContext() and getClient() reach the runtime from inside the call, so neither is an argument. The trailing options object carries this task's own TaskSpec.

CODE
const extract = dag.task("extract", async () => 42);
const transform = dag.task(async function transform() {}); // id "transform"
task< TArgs extends void | object = void, TReturn = unknown, >( taskId: string, handler: (args: TArgs) => TReturn | Promise<TReturn>, options?: TaskSpec, ): TaskFactory<TArgs, TReturn>

Type Parameters

  • TArgs extends void | object = void
  • TReturn = unknown

Parameters

  • taskId (string)
  • handler ((args: TArgs) => TReturn | Promise<TReturn>)
  • options (TaskSpec, optional)

Returns

  • TaskFactory<TArgs, TReturn>
task< TArgs extends void | object = void, TReturn = unknown, >( handler: (args: TArgs) => TReturn | Promise<TReturn>, options?: TaskSpec, ): TaskFactory<TArgs, TReturn>

Declare a task whose id is the handler's function name.

airflow-ts-pack keeps handler names intact, so minification cannot change a task id. An anonymous handler has no name to take one from, and needs TaskSpec.taskId to give it one.

Type Parameters

  • TArgs extends void | object = void
  • TReturn = unknown

Parameters

  • handler ((args: TArgs) => TReturn | Promise<TReturn>)
  • options (TaskSpec, optional)

Returns

  • TaskFactory<TArgs, TReturn>
task< TArgs extends void | object = void, TResult = unknown, >( taskId: string, operator: Operator<TArgs, TResult>, options?: TaskSpec, ): TaskFactory<TArgs, TResult>

Declare a task an operator runs, such as triggerDagRun(spec). Its inputs and result are the operator's own. It has no handler to take an id from, so the id is required.

Type Parameters

  • TArgs extends void | object = void
  • TResult = unknown

Parameters

  • taskId (string)
  • operator (Operator<TArgs, TResult>)
  • options (TaskSpec, optional)

Returns

  • TaskFactory<TArgs, TResult>
task< TArgs extends void | object = void, TResult = unknown, >( operator: Operator<TArgs, TResult>, options: TaskSpec, ): TaskFactory<TArgs, TResult>

Declare a task of this Dag, and return the factory that places it.

A handler takes one object of named arguments, and every argument in it becomes an input of the returned TaskFactory; getContext() and getClient() reach the runtime from inside the call, so neither is an argument. The trailing options object carries this task's own TaskSpec.

CODE
const extract = dag.task("extract", async () => 42);
const transform = dag.task(async function transform() {}); // id "transform"

Type Parameters

  • TArgs extends void | object = void
  • TResult = unknown

Parameters

  • operator (Operator<TArgs, TResult>)
  • options (TaskSpec)

Returns

  • TaskFactory<TArgs, TResult>

taskGroup( groupId: string, options?: TaskGroupOptions, ): TaskGroupRef

Declare a task group of this Dag.

The group prefixes the id of everything declared in it, unless TaskGroupOptions.prefixGroupId turns that off, and stands at either end of an order-only edge in its own right.

CODE
const staging = dag.taskGroup("staging"); // tasks "staging.<id>"
const checks = dag.taskGroup("checks", { prefixGroupId: false }); // ids as written

Parameters

  • groupId (string)
  • options (TaskGroupOptions, optional)

Returns

  • TaskGroupRef