Proposal for firestorm garbage collector (FGC) fixes...
Example data...
A simple example is a user ("user7") document changing his location field from "CA" to "VA". The KVStore will then have TermFreqRow records that would look like...
't'/1/CA/user7/10
't'/1/WA/user7/11
Above, 1 is a fieldId for the location field. "user7" is the docId. 10 and 11 are docNumbers.
The problem...
newFirestormReader() currently, roughly, looks like...
r = f.store.Reader() // A
s = f.compensator.Snapshot() // B
That order of operations, we've discussed, is backwards, as concurrent indexers might inadvertently increase the snapshot's maxRead and inFlight in between step A and step B, leaving the firestormReader unknowingly with an outdated, incorrect r.
The proposed fix (fix 1) to that was to reverse the steps...
s = compensator.Snapshot() // B
r = f.store.Reader() // A
And, additionally, (fix 2) we discussed adding locking to the newFirestormReader()...
X.lock()
s = compensator.Snapshot() // B
r = f.store.Reader() // A
X.unlock()
The FGC would have to be changed to respect the X locking, to prevent a concurrent FGC from incorrectly deleting records (like the "CA" record) in between steps B and A.
Aside: indexers might be able to ignore the proposed X lock, and perhaps just be able to write their new records at will, but that optimization idea won't be discussed here.
If f.store.Reader() creation (step A) is fast, then we're done, and the proposals so far seem to lead us to a correct FGC implementation, since any deletions by the FGC won't been seen by a slow reader since the slow reader has its own, stable snapshot that never sees later deletions.
The problem...
The worry is that f.store.Reader() creation (operation A) might be slow.
...
At this point, I was going to delve into writing down the complex proposal (proposal Q) of ordering the firestormReaders into a doubly-linked list, with ref-counting, so that FGC would only operate with the oldest, unused firestormReader in its scanning...
...but, I think now I should just do the above mini fixes, and experiment on their impact with rocksdb-firestorm.
And, if turns out we need to do proposal Q after those experiments, then write this out later.