sdk/org.apache.airflow.sdk/Airflow

Airflow

Container for the annotation-based Dag-authoring API.

Annotating a class with Dag generates a <Class>Builder whose static build() returns the DagDef to add to a Bundle, and a <Class>Deps wiring view for the class's Deps class to implement.

Example:

@Airflow.Dag(id = "my_pipeline", schedule = "@daily")
public class MyPipeline {

    @Airflow.Task(id = "extract", retries = 2)
    public long extract(Client client) { ... }

    @Airflow.Task(id = "transform")
    public long transform(Client client, long extracted) { ... }

    @Airflow.Deps
    static class Wiring implements MyPipelineDeps {
      void depends() { transform(extract()); }
    }
}

A task method's data parameters, meaning every parameter other than the injected Client and Context, receive by position the inputs the Deps class wired. For a task the Python Dag file declares with @task.stub, the arguments bound at that call site take their place. Keyword arguments bind by name instead through a single TaskInput parameter.

Types

Dag

@Target(allowedTargets = [AnnotationTarget.CLASS])

annotation class Dag(val id: String = "", val to: String = "", val schedule: String = "", val queue: String = "", val description: String = "", val dagDisplayName: String = "", val docMd: String = "", val startDate: String = "", val endDate: String = "", val dagrunTimeout: String = "", val tags: Array<String> = [], val maxActiveTasks: Int = -1, val maxActiveRuns: Int = -1, val maxConsecutiveFailedDagRuns: Int = -1, val catchup: Boolean = false, val failFast: Boolean = false, val renderTemplateAsNativeObj: Boolean = false, val disableBundleVersioning: Boolean = false, val isPausedUponCreation: Boolean = false)

Annotation to automate a Dag-builder pattern.

Deps

@Target(allowedTargets = [AnnotationTarget.CLASS])

annotation class Deps

Marks the nested class that declares this Dag's task graph.

@Target(allowedTargets = [AnnotationTarget.FUNCTION])

annotation class If(val id: String = "", val owner: String = "airflow", val startDate: String = "", val endDate: String = "", val triggerRule: String = "all_success", val dependsOnPast: Boolean = false, val ignoreFirstDependsOnPast: Boolean = false, val waitForPastDependsBeforeSkipping: Boolean = false, val waitForDownstream: Boolean = false, val retries: Int = 0, val queue: String = "default", val pool: String = "default_pool", val poolSlots: Int = 1, val executionTimeout: String = "", val retryDelay: String = "PT5M", val retryExponentialBackoff: Double = 0.0, val maxRetryDelay: String = "", val priorityWeight: Int = 1, val weightRule: String = "downstream", val executor: String = "", val doXcomPush: Boolean = true, val emailOnFailure: Boolean = true, val emailOnRetry: Boolean = true, val docMd: String = "", val mapIndexTemplate: String = "", val maxActiveTisPerDag: Int = -1, val maxActiveTisPerDagrun: Int = -1)

Marks a task method whose boolean picks one of two tasks; the other is skipped.

Switch

@Target(allowedTargets = [AnnotationTarget.FUNCTION])

annotation class Switch(val id: String = "", val owner: String = "airflow", val startDate: String = "", val endDate: String = "", val triggerRule: String = "all_success", val dependsOnPast: Boolean = false, val ignoreFirstDependsOnPast: Boolean = false, val waitForPastDependsBeforeSkipping: Boolean = false, val waitForDownstream: Boolean = false, val retries: Int = 0, val queue: String = "default", val pool: String = "default_pool", val poolSlots: Int = 1, val executionTimeout: String = "", val retryDelay: String = "PT5M", val retryExponentialBackoff: Double = 0.0, val maxRetryDelay: String = "", val priorityWeight: Int = 1, val weightRule: String = "downstream", val executor: String = "", val doXcomPush: Boolean = true, val emailOnFailure: Boolean = true, val emailOnRetry: Boolean = true, val docMd: String = "", val mapIndexTemplate: String = "", val maxActiveTisPerDag: Int = -1, val maxActiveTisPerDagrun: Int = -1)

Marks a task method that chooses one of several tasks to run; every other one is skipped.

Task

@Target(allowedTargets = [AnnotationTarget.FUNCTION])

annotation class Task(val id: String = "", val owner: String = "airflow", val startDate: String = "", val endDate: String = "", val triggerRule: String = "all_success", val dependsOnPast: Boolean = false, val ignoreFirstDependsOnPast: Boolean = false, val waitForPastDependsBeforeSkipping: Boolean = false, val waitForDownstream: Boolean = false, val retries: Int = 0, val queue: String = "default", val pool: String = "default_pool", val poolSlots: Int = 1, val executionTimeout: String = "", val retryDelay: String = "PT5M", val retryExponentialBackoff: Double = 0.0, val maxRetryDelay: String = "", val priorityWeight: Int = 1, val weightRule: String = "downstream", val executor: String = "", val doXcomPush: Boolean = true, val emailOnFailure: Boolean = true, val emailOnRetry: Boolean = true, val docMd: String = "", val mapIndexTemplate: String = "", val maxActiveTisPerDag: Int = -1, val maxActiveTisPerDagrun: Int = -1)

Annotation to automate task definition in a Dag-builder pattern.

TaskGroup

@Target(allowedTargets = [AnnotationTarget.CLASS])

annotation class TaskGroup(val id: String = "")

Marks a nested class that groups the tasks declared inside it, as Python's TaskGroup does.

TaskHandler

@Target(allowedTargets = [AnnotationTarget.FUNCTION])

annotation class TaskHandler(val dag: String, val task: String = "")

Marks a method as the Java body of a task the Python Dag file declares with @task.stub.