BashOperator¶

Use the BashOperator to execute commands in a Bash shell.

airflow/example_dags/example_bash_operator.pyView Source

run_this = BashOperator(
    task_id='run_after_loop',
    bash_command='echo 1',
    dag=dag,
)

Templating¶

You can use Jinja templates to parameterize the bash_command argument.

airflow/example_dags/example_bash_operator.pyView Source

also_run_this = BashOperator(
    task_id='also_run_this',
    bash_command='echo "run_id={{ run_id }} | dag_run={{ dag_run }}"',
    dag=dag,
)

Warning

Care should be taken with “user” input or when using Jinja templates in the bash_command, as this bash operator does not perform any escaping or sanitization of the command.

This applies mostly to using “dag_run” conf, as that can be submitted via users in the Web UI. Most of the default template variables are not at risk.

For example, do not do this:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command='echo "Here is the message: \'{{ dag_run.conf["message"] if dag_run else "" }}\'"',
)

Instead, you should pass this via the env kwarg and use double-quotes inside the bash_command, as below:

bash_task = BashOperator(
    task_id="bash_task",
    bash_command='echo "here is the message: \'$message\'"',
    env={'message': '{{ dag_run.conf["message"] if dag_run else "" }}'},
)

Troubleshooting¶

Jinja template not found¶

Add a space after the script name when directly calling a Bash script with the bash_command argument. This is because Airflow tries to apply a Jinja template to it, which will fail.

t2 = BashOperator(
    task_id='bash_example',

    # This fails with 'Jinja template not found' error
    # bash_command="/home/batcher/test.sh",

    # This works (has a space after)
    bash_command="/home/batcher/test.sh ",
    dag=dag)

However, if you want to use templating in your bash script, do not add the space and instead put your bash script in a location relative to the directory containing the DAG file. So if your DAG file is in /usr/local/airflow/dags/test_dag.py, you can move your test.sh file to any location under /usr/local/airflow/dags/ (Example: /usr/local/airflow/dags/scripts/test.sh) and pass the relative path to bash_command as shown below:

t2 = BashOperator(
    task_id='bash_example',
    # "scripts" folder is under "/usr/local/airflow/dags"
    bash_command="scripts/test.sh",
    dag=dag)

Creating separate folder for bash scripts may be desirable for many reasons, like separating your script’s logic and pipeline code, allowing for proper code highlighting in files composed in different languages, and general flexibility in structuring pipelines.

It is also possible to define your template_searchpath as pointing to any folder locations in the DAG constructor call.

Example:

dag = DAG("example_bash_dag", template_searchpath="/opt/scripts")
t2 = BashOperator(
    task_id='bash_example',
    # "test.sh" is a file under "/opt/scripts"
    bash_command="test.sh ",
    dag=dag)