Parseable is experiencing data loss due to a time skew in local-sync. The scheduler operates on minute-based(eventually configurable) intervals and performs flush operations at the end of each interval. However, a timing discrepancy causes late-arriving data to be appended after EOF markers, rendering this data kinda invisible to readers.
The scheduler triggers a flush operation at the end of each minute interval
The flush operation:
- Writes all in-memory buffered data to disk
- Appends an EOF marker to signal completion
Due to time skew, additional data from the current minute continues to arrive after the EOF marker has been written and gets appended(ref). Reader processes stop reading when they encounter the EOF marker, ignoring any subsequent data.
The fundamental issue is a race condition between:
- Time skew introduced by scheduler
- The actual ingestion timeline
This skew creates a scenario where the scheduler prematurely signals completion (via EOF) while logs belonging to the same time window continue to arrive, this is mainly because we don't care to check if the file being flushed is from the current minute or not(ref).
Data loss: All records written after the EOF marker are effectively lost. Potential for silent failures as the system appears to function correctly after dataloss
Don't flush/finish if data contained is of current minute, as solved by is_current