Skip to content

Instantly share code, notes, and snippets.

@mumrah
Last active July 13, 2016 15:13
Show Gist options
  • Save mumrah/4627896 to your computer and use it in GitHub Desktop.
Save mumrah/4627896 to your computer and use it in GitHub Desktop.
Example of batching and multi-threading Solr updates
public static class SolrUpdater implements Runnable {
private final UpdateRequest req = new UpdateRequest();
private final SolrServer solr;
private final BlockingQueue<String> strings;
private final AtomicLong id;
private final int batchSize = 100;
private volatile int batchedUpdates = 0;
public SolrUpdater(SolrServer solr, BlockingQueue<String> strings,
AtomicLong id) {
this.solr = solr;
this.strings = strings;
this.id = id;
}
@Override
public void run() {
while(true) {
String text = strings.take();
SolrInputDocument doc = new SolrInputDocument();
doc.setField("id", "doc-" + id.getAndIncrement());
doc.setField("text", text);
req.add(doc);
batchedUpdates++;
if(batchedUpdates >= batchSize) {
req.process(solr);
req.clear();
}
}
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment