airflow.providers.amazon.aws.transfers.mongo_to_s3
¶
Module Contents¶
-
class
airflow.providers.amazon.aws.transfers.mongo_to_s3.
MongoToS3Operator
(*, mongo_conn_id: str, s3_conn_id: str, mongo_collection: str, mongo_query: Union[list, dict], s3_bucket: str, s3_key: str, mongo_db: Optional[str] = None, replace: bool = False, **kwargs)[source]¶ Bases:
airflow.models.BaseOperator
Mongo -> S3 A more specific baseOperator meant to move data from mongo via pymongo to s3 via boto
- things to note
.execute() is written to depend on .transform() .transform() is meant to be extended by child classes to perform transformations unique to those operators needs
-
static
_stringify
(iterable: Iterable, joinable: str = '\n')[source]¶ Takes an iterable (pymongo Cursor or Array) containing dictionaries and returns a stringified version using python join
-
static
transform
(docs: Any)[source]¶ - Processes pyMongo cursor and returns an iterable with each element being
a JSON serializable dictionary
Base transform() assumes no processing is needed ie. docs is a pyMongo cursor of documents and cursor just needs to be passed through
Override this method for custom transformations