airflow.providers.common.sql.datafusion.object_storage_provider

Classes

S3ObjectStorageProvider

S3 Object Storage Provider using DataFusion's AmazonS3.

LocalObjectStorageProvider

Local Object Storage Provider using DataFusion's LocalFileSystem.

Functions

get_object_storage_provider(storage_type)

Get an object storage provider based on the storage type.

Module Contents

class airflow.providers.common.sql.datafusion.object_storage_provider.S3ObjectStorageProvider(context=None)[source]

Bases: airflow.providers.common.sql.datafusion.base.ObjectStorageProvider

S3 Object Storage Provider using DataFusion’s AmazonS3.

property get_storage_type: airflow.providers.common.sql.config.StorageType[source]

Return the storage type.

create_object_store(path, connection_config=None)[source]

Create an S3 object store using DataFusion’s AmazonS3.

get_scheme()[source]

Return the scheme for S3.

class airflow.providers.common.sql.datafusion.object_storage_provider.LocalObjectStorageProvider(context=None)[source]

Bases: airflow.providers.common.sql.datafusion.base.ObjectStorageProvider

Local Object Storage Provider using DataFusion’s LocalFileSystem.

property get_storage_type: airflow.providers.common.sql.config.StorageType[source]

Return the storage type.

create_object_store(path, connection_config=None)[source]

Create a Local object store.

get_scheme()[source]

Return the scheme to a Local file system.

airflow.providers.common.sql.datafusion.object_storage_provider.get_object_storage_provider(storage_type)[source]

Get an object storage provider based on the storage type.

Was this entry helpful?