airflow.contrib.hooks.gcp_pubsub_hook

Module Contents

airflow.contrib.hooks.gcp_pubsub_hook._format_subscription(project, subscription)

airflow.contrib.hooks.gcp_pubsub_hook._format_topic(project, topic)

exception airflow.contrib.hooks.gcp_pubsub_hook.PubSubException

Bases: Exception

class airflow.contrib.hooks.gcp_pubsub_hook.PubSubHook(gcp_conn_id='google_cloud_default', delegate_to=None)

Bases: airflow.contrib.hooks.gcp_api_base_hook.GoogleCloudBaseHook

Hook for accessing Google Pub/Sub.

The GCP project against which actions are applied is determined by the project embedded in the Connection referenced by gcp_conn_id.

get_conn(self)

Returns a Pub/Sub service object.

Return type

googleapiclient.discovery.Resource

publish(self, project, topic, messages)

Publishes messages to a Pub/Sub topic.

Parameters

• project (str) – the GCP project ID in which to publish

• topic (str) – the Pub/Sub topic to which to publish; do not include the projects/{project}/topics/ prefix.

• messages (list of PubSub messages; see http://cloud.google.com/pubsub/docs/reference/rest/v1/PubsubMessage) – messages to publish; if the data field in a message is set, it should already be base64 encoded.

create_topic(self, project, topic, fail_if_exists=False)

Creates a Pub/Sub topic, if it does not already exist.

Parameters

• project (str) – the GCP project ID in which to create the topic

• topic (str) – the Pub/Sub topic name to create; do not include the projects/{project}/topics/ prefix.

• fail_if_exists (bool) – if set, raise an exception if the topic already exists

delete_topic(self, project, topic, fail_if_not_exists=False)

Deletes a Pub/Sub topic if it exists.

Parameters

• project (str) – the GCP project ID in which to delete the topic

• topic (str) – the Pub/Sub topic name to delete; do not include the projects/{project}/topics/ prefix.

• fail_if_not_exists (bool) – if set, raise an exception if the topic does not exist

create_subscription(self, topic_project, topic, subscription=None, subscription_project=None, ack_deadline_secs=10, fail_if_exists=False)

Creates a Pub/Sub subscription, if it does not already exist.

Parameters

• topic_project (str) – the GCP project ID of the topic that the subscription will be bound to.

• topic (str) – the Pub/Sub topic name that the subscription will be bound to create; do not include the projects/{project}/subscriptions/ prefix.

• subscription (str) – the Pub/Sub subscription name. If empty, a random name will be generated using the uuid module

• subscription_project (str) – the GCP project ID where the subscription will be created. If unspecified, topic_project will be used.

• ack_deadline_secs (int) – Number of seconds that a subscriber has to acknowledge each message pulled from the subscription

• fail_if_exists (bool) – if set, raise an exception if the topic already exists

Returns

subscription name which will be the system-generated value if the subscription parameter is not supplied

Return type

str

delete_subscription(self, project, subscription, fail_if_not_exists=False)

Deletes a Pub/Sub subscription, if it exists.

Parameters

• project (str) – the GCP project ID where the subscription exists

• subscription (str) – the Pub/Sub subscription name to delete; do not include the projects/{project}/subscriptions/ prefix.

• fail_if_not_exists (bool) – if set, raise an exception if the topic does not exist

pull(self, project, subscription, max_messages, return_immediately=False)

Pulls up to max_messages messages from Pub/Sub subscription.

Parameters

• project (str) – the GCP project ID where the subscription exists

• subscription (str) – the Pub/Sub subscription name to pull from; do not include the ‘projects/{project}/topics/’ prefix.

• max_messages (int) – The maximum number of messages to return from the Pub/Sub API.

• return_immediately (bool) – If set, the Pub/Sub API will immediately return if no messages are available. Otherwise, the request will block for an undisclosed, but bounded period of time

Returns

A list of Pub/Sub ReceivedMessage objects each containing an ackId property and a message property, which includes the base64-encoded message content. See https://cloud.google.com/pubsub/docs/reference/rest/v1/projects.subscriptions/pull#ReceivedMessage

acknowledge(self, project, subscription, ack_ids)

Pulls up to max_messages messages from Pub/Sub subscription.

Parameters

• project (str) – the GCP project name or ID in which to create the topic

• subscription (str) – the Pub/Sub subscription name to delete; do not include the ‘projects/{project}/topics/’ prefix.

• ack_ids (list) – List of ReceivedMessage ackIds from a previous pull response