clj-headlights.pubsub
Helpers to use Pub/Sub in DataFlow jobs
pubsub-url->type
(pubsub-url->type url)Inputs: [url :- s/Str] Returns: s/Str
Extract the type from a PubSub URL, e.g. topics in projects/my-project/topics/my-topic
read-stream
(read-stream url)Inputs: [url :- (s/maybe s/Str)] Returns: (s/maybe PTransform)
Create a PTransform that reads either a Pub/Sub topic or subscription, based on the given URL.
read-stream-with-ts
(read-stream-with-ts url)(read-stream-with-ts url attr-name)Inputs: ([url :- (s/maybe s/Str)] [url :- (s/maybe s/Str) attr-name :- s/Str]) Returns: (s/maybe PTransform)
Create a PTransform that reads a Pub/Sub topic or subscription, based on the given URL. Uses .timestampLabel to assign timestamps to each message. Defaults to looking up the ‘ts’ attribute if no other name is provided.