clj-headlights.bigquery-io

Common bigquery utilites for clojure dataflow jobs.

CreateDispositions

read-from-query

(read-from-query query)

Inputs: [query :- s/Str] Returns: PTransform

read-from-query-and-extract

(read-from-query-and-extract pipeline name query field)

Inputs: [pipeline :- pcollections/PCollectionType name :- s/Str query :- s/Str field :- s/Str] Returns: pcollections/PCollectionType

schema-field->bq-field

(schema-field->bq-field schema-field)

table-row-maker

(table-row-maker row-as-map schema)

to-tablerow

(to-tablerow str)

write

(write output pcoll)

Inputs: [output :- (s/cond-pre s/Str SerializableFunction) pcoll :- pcollections/PCollectionType] Returns: pcollections/PCollectionType

Write elements of a pcoll to bigquery in json format

write-with-schema

(write-with-schema pcoll name output schema write-options)

Inputs: [pcoll :- pcollections/PCollectionType name :- s/Str output :- (s/cond-pre s/Str SerializableFunction) schema :- [{:name s/Str, :type s/Str, (s/optional-key :mode) s/Str}] write-options :- {s/Keyword s/Any}] Returns: pcollections/PCollectionType

Write elements of a pcoll to bigquery in json format

WriteDispostions