Skip to content

Instantly share code, notes, and snippets.

@kornypoet
Last active December 19, 2015 15:48
Show Gist options
  • Save kornypoet/5978578 to your computer and use it in GitHub Desktop.
Save kornypoet/5978578 to your computer and use it in GitHub Desktop.
Suggested setup for an S3 trident spout
class S3Spout {
class S3Client { // Travis will write
// Make using S3 easy like a boss
}
private S3Client _client; // Travis will write
class Coordinator { // Eric will write
// Because Java, you have access to _client in here for use
public String getNext() {
// figure out what file is next
return "foo";
}
}
private Coordinator _coord; // Eric will write
// Storm batch spout interface stuff goes here
@Override
public void ack(long batchid) {
// Travis will write
}
// ... Travis will write
@Override
public void(long batchid, TridentCollector, collector) {
File nextFile = _coord.getNext();
for (Chunk c: _client.confirmDownloadInChunks()) {
collector.emit(c.toTuple();
}
_coord.confirmFinished(nextFile);
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment