Skip to content

Instantly share code, notes, and snippets.

@polleyg
Created January 23, 2019 09:12
Show Gist options
  • Save polleyg/26557b8b8b8f9d1974535872ae7f58bb to your computer and use it in GitHub Desktop.
Save polleyg/26557b8b8b8f9d1974535872ae7f58bb to your computer and use it in GitHub Desktop.
beam_sql_part_4
[..]
.apply("transform_to_string", ParDo.of(new RowToString()))
.apply("write_to_gcs", TextIO.write().to("gs://batch-pipeline-sql/output/output.csv").withoutSharding());
[..]
//ParDo for Row (SQL) -> String
public static class RowToString extends DoFn<Row, String> {
@ProcessElement
public void processElement(ProcessContext c) {
String line = c.element().getValues()
.stream()
.map(Object::toString)
.collect(Collectors.joining(","));
c.output(line);
}
}
[..]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment