airflow.providers.amazon.aws.operators.datasync
¶
Create, get, update, execute and delete an AWS DataSync Task.
Module Contents¶
Classes¶
Find, Create, Update, Execute and Delete AWS DataSync Tasks. |
|
This operator is deprecated. |
- class airflow.providers.amazon.aws.operators.datasync.DataSyncOperator(*, aws_conn_id='aws_default', wait_interval_seconds=30, max_iterations=60, task_arn=None, source_location_uri=None, destination_location_uri=None, allow_random_task_choice=False, allow_random_location_choice=False, create_task_kwargs=None, create_source_location_kwargs=None, create_destination_location_kwargs=None, update_task_kwargs=None, task_execution_kwargs=None, delete_task_after_execution=False, **kwargs)[source]¶
Bases:
airflow.models.BaseOperator
Find, Create, Update, Execute and Delete AWS DataSync Tasks.
If
do_xcom_push
is True, then the DataSync TaskArn and TaskExecutionArn which were executed will be pushed to an XCom.See also
For more information on how to use this operator, take a look at the guide: Interact with AWS DataSync Tasks
Note
There may be 0, 1, or many existing DataSync Tasks defined in your AWS environment. The default behavior is to create a new Task if there are 0, or execute the Task if there was 1 Task, or fail if there were many Tasks.
- Parameters
aws_conn_id (str) – AWS connection to use.
wait_interval_seconds (int) – Time to wait between two consecutive calls to check TaskExecution status.
max_iterations (int) – Maximum number of consecutive calls to check TaskExecution status.
task_arn (Optional[str]) – AWS DataSync TaskArn to use. If None, then this operator will attempt to either search for an existing Task or attempt to create a new Task.
source_location_uri (Optional[str]) – Source location URI to search for. All DataSync Tasks with a LocationArn with this URI will be considered. Example:
smb://server/subdir
destination_location_uri (Optional[str]) – Destination location URI to search for. All DataSync Tasks with a LocationArn with this URI will be considered. Example:
s3://airflow_bucket/stuff
allow_random_task_choice (bool) – If multiple Tasks match, one must be chosen to execute. If allow_random_task_choice is True then a random one is chosen.
allow_random_location_choice (bool) – If multiple Locations match, one must be chosen when creating a task. If allow_random_location_choice is True then a random one is chosen.
create_task_kwargs (Optional[dict]) – If no suitable TaskArn is identified, it will be created if
create_task_kwargs
is defined.create_task_kwargs
is then used internally like this:boto3.create_task(**create_task_kwargs)
Example:{'Name': 'xyz', 'Options': ..., 'Excludes': ..., 'Tags': ...}
create_source_location_kwargs (Optional[dict]) – If no suitable LocationArn is found, a Location will be created if
create_source_location_kwargs
is defined.create_source_location_kwargs
is then used internally like this:boto3.create_location_xyz(**create_source_location_kwargs)
The xyz is determined from the prefix of source_location_uri, egsmb:/...
ors3:/...
Example:{'Subdirectory': ..., 'ServerHostname': ..., ...}
create_destination_location_kwargs (Optional[dict]) – If no suitable LocationArn is found, a Location will be created if
create_destination_location_kwargs
is defined.create_destination_location_kwargs
is used internally like this:boto3.create_location_xyz(**create_destination_location_kwargs)
The xyz is determined from the prefix of destination_location_uri, egsmb:/...` or ``s3:/...
Example:{'S3BucketArn': ..., 'S3Config': {'BucketAccessRoleArn': ...}, ...}
update_task_kwargs (Optional[dict]) – If a suitable TaskArn is found or created, it will be updated if
update_task_kwargs
is defined.update_task_kwargs
is used internally like this:boto3.update_task(TaskArn=task_arn, **update_task_kwargs)
Example:{'Name': 'xyz', 'Options': ..., 'Excludes': ...}
task_execution_kwargs (Optional[dict]) – Additional kwargs passed directly when starting the Task execution, used internally like this:
boto3.start_task_execution(TaskArn=task_arn, **task_execution_kwargs)
delete_task_after_execution (bool) – If True then the TaskArn which was executed will be deleted from AWS DataSync on successful completion.
- Raises
AirflowException – If
task_arn
was not specified, or if eithersource_location_uri
ordestination_location_uri
were not specified.AirflowException – If source or destination Location were not found and could not be created.
AirflowException – If
choose_task
orchoose_location
fails.AirflowException – If Task creation, update, execution or delete fails.
- template_fields :Sequence[str] = ['task_arn', 'source_location_uri', 'destination_location_uri', 'create_task_kwargs',...[source]¶
- get_hook(self)[source]¶
Create and return DataSyncHook.
- Return DataSyncHook
An DataSyncHook instance.
- Return type
- class airflow.providers.amazon.aws.operators.datasync.AWSDataSyncOperator(*args, **kwargs)[source]¶
Bases:
DataSyncOperator
This operator is deprecated. Please use
airflow.providers.amazon.aws.operators.datasync.DataSyncOperator
.