Skip to content

Instantly share code, notes, and snippets.

@daynebatten
Created March 21, 2016 18:26
Show Gist options
  • Save daynebatten/6732a8b37c468e07f7d2 to your computer and use it in GitHub Desktop.
Save daynebatten/6732a8b37c468e07f7d2 to your computer and use it in GitHub Desktop.
while (getFilePosition() <= end || in.needAdditionalRecordAfterSplit()) {
try {
if (pos == 0) {
newSize = skipUtfByteOrderMark();
} else {
newSize = in.readLine(value, maxLineLength, maxBytesToConsume(pos));
pos += newSize;
}
if ((newSize == 0) || (newSize < maxLineLength)) {
break;
}
// line too long. try again
LOG.info("Skipped line of size " + newSize + " at pos " +
(pos - newSize));
} catch(IOException e) {
fileBroken = true;
LOG.warn("Problem reading file " + this.splitName + " - " + e.getMessage());
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment