airflow.providers.google.cloud.hooks.pubsub

This module contains a Google Pub/Sub Hook.

Module Contents

Classes

PubSubHook

Hook for accessing Google Pub/Sub.

PubSubAsyncHook

Class to get asynchronous hook for Google Cloud PubSub.
exception airflow.providers.google.cloud.hooks.pubsub.PubSubException[source]

Bases: Exception

Alias for Exception.

class airflow.providers.google.cloud.hooks.pubsub.PubSubHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for accessing Google Pub/Sub.

The Google Cloud project against which actions are applied is determined by the project embedded in the Connection referenced by gcp_conn_id.

get_conn()[source]

Retrieve connection to Google Cloud Pub/Sub.

Returns

Google Cloud Pub/Sub client object.

Return type

google.cloud.pubsub_v1.PublisherClient

subscriber_client()[source]

Creates SubscriberClient.

Returns

Google Cloud Pub/Sub client object.

Return type

google.cloud.pubsub_v1.SubscriberClient

publish(topic, messages, project_id=PROVIDE_PROJECT_ID)[source]

Publish messages to a Pub/Sub topic.

Parameters

  • topic (str) – the Pub/Sub topic to which to publish; do not include the projects/{project}/topics/ prefix.

  • messages (list[dict]) – messages to publish; if the data field in a message is set, it should be a bytestring (utf-8 encoded) https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#pubsubmessage

  • project_id (str) – Optional, the Google Cloud project ID in which to publish. If set to None or missing, the default project_id from the Google Cloud connection is used.

create_topic(topic, project_id=PROVIDE_PROJECT_ID, fail_if_exists=False, labels=None, message_storage_policy=None, kms_key_name=None, schema_settings=None, message_retention_duration=None, retry=DEFAULT, timeout=None, metadata=())[source]

Create a Pub/Sub topic, if it does not already exist.

Parameters

  • topic (str) – the Pub/Sub topic name to create; do not include the projects/{project}/topics/ prefix.

  • project_id (str) – Optional, the Google Cloud project ID in which to create the topic If set to None or missing, the default project_id from the Google Cloud connection is used.

  • fail_if_exists (bool) – if set, raise an exception if the topic already exists

  • labels (dict[str, str] | None) – Client-assigned labels; see https://cloud.google.com/pubsub/docs/labels

  • message_storage_policy (dict | google.cloud.pubsub_v1.types.MessageStoragePolicy) – Policy constraining the set of Google Cloud regions where messages published to the topic may be stored. If not present, then no constraints are in effect. Union[dict, google.cloud.pubsub_v1.types.MessageStoragePolicy]

  • kms_key_name (str | None) – The resource name of the Cloud KMS CryptoKey to be used to protect access to messages published on this topic. The expected format is projects/*/locations/*/keyRings/*/cryptoKeys/*.

  • schema_settings (dict | google.cloud.pubsub_v1.types.SchemaSettings) – (Optional) Settings for validating messages published against an existing schema. The expected format is projects/*/schemas/*.

  • message_retention_duration (str | None) – (Optional) Indicates the minimum duration to retain a message after it is published to the topic. The expected format is a duration in seconds with up to nine fractional digits, ending with ‘s’. Example: “3.5s”.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

delete_topic(topic, project_id=PROVIDE_PROJECT_ID, fail_if_not_exists=False, retry=DEFAULT, timeout=None, metadata=())[source]

Delete a Pub/Sub topic if it exists.

Parameters

  • topic (str) – the Pub/Sub topic name to delete; do not include the projects/{project}/topics/ prefix.

  • project_id (str) – Optional, the Google Cloud project ID in which to delete the topic. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • fail_if_not_exists (bool) – if set, raise an exception if the topic does not exist

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

create_subscription(topic, project_id=PROVIDE_PROJECT_ID, subscription=None, subscription_project_id=None, ack_deadline_secs=10, fail_if_exists=False, push_config=None, retain_acked_messages=None, message_retention_duration=None, labels=None, enable_message_ordering=False, expiration_policy=None, filter_=None, dead_letter_policy=None, retry_policy=None, retry=DEFAULT, timeout=None, metadata=())[source]

Create a Pub/Sub subscription, if it does not already exist.

Parameters

  • topic (str) – the Pub/Sub topic name that the subscription will be bound to create; do not include the projects/{project}/subscriptions/ prefix.

  • project_id (str) – Optional, the Google Cloud project ID of the topic that the subscription will be bound to. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • subscription (str | None) – the Pub/Sub subscription name. If empty, a random name will be generated using the uuid module

  • subscription_project_id (str | None) – the Google Cloud project ID where the subscription will be created. If unspecified, project_id will be used.

  • ack_deadline_secs (int) – Number of seconds that a subscriber has to acknowledge each message pulled from the subscription

  • fail_if_exists (bool) – if set, raise an exception if the topic already exists

  • push_config (dict | google.cloud.pubsub_v1.types.PushConfig | None) – If push delivery is used with this subscription, this field is used to configure it. An empty pushConfig signifies that the subscriber will pull and ack messages using API methods.

  • retain_acked_messages (bool | None) – Indicates whether to retain acknowledged messages. If true, then messages are not expunged from the subscription’s backlog, even if they are acknowledged, until they fall out of the message_retention_duration window. This must be true if you would like to Seek to a timestamp.

  • message_retention_duration (dict | google.cloud.pubsub_v1.types.Duration | None) – How long to retain unacknowledged messages in the subscription’s backlog, from the moment a message is published. If retain_acked_messages is true, then this also configures the retention of acknowledged messages, and thus configures how far back in time a Seek can be done. Defaults to 7 days. Cannot be more than 7 days or less than 10 minutes.

  • labels (dict[str, str] | None) – Client-assigned labels; see https://cloud.google.com/pubsub/docs/labels

  • enable_message_ordering (bool) – If true, messages published with the same ordering_key in PubsubMessage will be delivered to the subscribers in the order in which they are received by the Pub/Sub system. Otherwise, they may be delivered in any order.

  • expiration_policy (dict | google.cloud.pubsub_v1.types.ExpirationPolicy | None) – A policy that specifies the conditions for this subscription’s expiration. A subscription is considered active as long as any connected subscriber is successfully consuming messages from the subscription or is issuing operations on the subscription. If expiration_policy is not set, a default policy with ttl of 31 days will be used. The minimum allowed value for expiration_policy.ttl is 1 day.

  • filter – An expression written in the Cloud Pub/Sub filter language. If non-empty, then only PubsubMessages whose attributes field matches the filter are delivered on this subscription. If empty, then no messages are filtered out.

  • dead_letter_policy (dict | google.cloud.pubsub_v1.types.DeadLetterPolicy | None) – A policy that specifies the conditions for dead lettering messages in this subscription. If dead_letter_policy is not set, dead lettering is disabled.

  • retry_policy (dict | google.cloud.pubsub_v1.types.RetryPolicy | None) – A policy that specifies how Pub/Sub retries message delivery for this subscription. If not set, the default retry policy is applied. This generally implies that messages will be retried as soon as possible for healthy subscribers. RetryPolicy will be triggered on NACKs or acknowledgement deadline exceeded events for a given message.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

Returns

subscription name which will be the system-generated value if the subscription parameter is not supplied

Return type

str

delete_subscription(subscription, project_id=PROVIDE_PROJECT_ID, fail_if_not_exists=False, retry=DEFAULT, timeout=None, metadata=())[source]

Delete a Pub/Sub subscription, if it exists.

Parameters

  • subscription (str) – the Pub/Sub subscription name to delete; do not include the projects/{project}/subscriptions/ prefix.

  • project_id (str) – Optional, the Google Cloud project ID where the subscription exists If set to None or missing, the default project_id from the Google Cloud connection is used.

  • fail_if_not_exists (bool) – if set, raise an exception if the topic does not exist

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

pull(subscription, max_messages, project_id=PROVIDE_PROJECT_ID, return_immediately=False, retry=DEFAULT, timeout=None, metadata=())[source]

Pull up to max_messages messages from Pub/Sub subscription.

Parameters

  • subscription (str) – the Pub/Sub subscription name to pull from; do not include the ‘projects/{project}/topics/’ prefix.

  • max_messages (int) – The maximum number of messages to return from the Pub/Sub API.

  • project_id (str) – Optional, the Google Cloud project ID where the subscription exists. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • return_immediately (bool) – If set, the Pub/Sub API will immediately return if no messages are available. Otherwise, the request will block for an undisclosed, but bounded period of time

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

Returns

A list of Pub/Sub ReceivedMessage objects each containing an ackId property and a message property, which includes the base64-encoded message content. See https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.ReceivedMessage

Return type

list[google.cloud.pubsub_v1.types.ReceivedMessage]

acknowledge(subscription, project_id, ack_ids=None, messages=None, retry=DEFAULT, timeout=None, metadata=())[source]

Acknowledges the messages associated with the ack_ids from Pub/Sub subscription.

Parameters

  • subscription (str) – the Pub/Sub subscription name to delete; do not include the ‘projects/{project}/topics/’ prefix.

  • ack_ids (list[str] | None) – List of ReceivedMessage ackIds from a previous pull response. Mutually exclusive with messages argument.

  • messages (list[google.cloud.pubsub_v1.types.ReceivedMessage] | None) – List of ReceivedMessage objects to acknowledge. Mutually exclusive with ack_ids argument.

  • project_id (str) – Optional, the Google Cloud project name or ID in which to create the topic If set to None or missing, the default project_id from the Google Cloud connection is used.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

class airflow.providers.google.cloud.hooks.pubsub.PubSubAsyncHook(project_id=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseAsyncHook

Class to get asynchronous hook for Google Cloud PubSub.

sync_hook_class[source]
async acknowledge(subscription, project_id, ack_ids=None, messages=None, retry=DEFAULT, timeout=None, metadata=())[source]

Acknowledges the messages associated with the ack_ids from Pub/Sub subscription.

Parameters

  • subscription (str) – the Pub/Sub subscription name to delete; do not include the ‘projects/{project}/topics/’ prefix.

  • ack_ids (list[str] | None) – List of ReceivedMessage ackIds from a previous pull response. Mutually exclusive with messages argument.

  • messages (list[google.cloud.pubsub_v1.types.ReceivedMessage] | None) – List of ReceivedMessage objects to acknowledge. Mutually exclusive with ack_ids argument.

  • project_id (str) – Optional, the Google Cloud project name or ID in which to create the topic If set to None or missing, the default project_id from the Google Cloud connection is used.

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

async pull(subscription, max_messages, project_id=PROVIDE_PROJECT_ID, return_immediately=False, retry=DEFAULT, timeout=None, metadata=())[source]

Pull up to max_messages messages from Pub/Sub subscription.

Parameters

  • subscription (str) – the Pub/Sub subscription name to pull from; do not include the ‘projects/{project}/topics/’ prefix.

  • max_messages (int) – The maximum number of messages to return from the Pub/Sub API.

  • project_id (str) – Optional, the Google Cloud project ID where the subscription exists. If set to None or missing, the default project_id from the Google Cloud connection is used.

  • return_immediately (bool) – If set, the Pub/Sub API will immediately return if no messages are available. Otherwise, the request will block for an undisclosed, but bounded period of time

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – (Optional) A retry object used to retry requests. If None is specified, requests will not be retried.

  • timeout (float | None) – (Optional) The amount of time, in seconds, to wait for the request to complete. Note that if retry is specified, the timeout applies to each individual attempt.

  • metadata (Sequence[tuple[str, str]]) – (Optional) Additional metadata that is provided to the method.

Returns

A list of Pub/Sub ReceivedMessage objects each containing an ackId property and a message property, which includes the base64-encoded message content. See https://cloud.google.com/pubsub/docs/reference/rpc/google.pubsub.v1#google.pubsub.v1.ReceivedMessage

Return type

list[google.cloud.pubsub_v1.types.ReceivedMessage]

Was this entry helpful?