Skip to content

Instantly share code, notes, and snippets.

@thrawn01
Created December 11, 2024 12:24
Show Gist options
  • Save thrawn01/d26226c1217c1b1668068c26e25239f8 to your computer and use it in GitHub Desktop.
Save thrawn01/d26226c1217c1b1668068c26e25239f8 to your computer and use it in GitHub Desktop.
Solving the WAL problem domain
// Separating the WAL implementation hides the state complexity currently in *DB
// behind a simple problem domain -- Write and retrieve entries from the log.
//
// An additional benefit is this allows a user the option to write the WAL to
// local disk, or some other durable storage. While still maintaining the
// memtables in the object store.
w := wal.New(wal.Config{
// Store could be S3, or local file system depending on how the
// user wants slateDB to operate.
Store: NewTableStore(bucket, sstable.Config{}, "/path"),
})
// Recovers any immutable pages flushed to disk due to crash if any?
if err := w.Recover(); err != nil {
panic(err)
}
// Operates on RowEntry to allow the user access to Sequence numbers used to
// implement transactions if needed.
if err := w.Put(types.RowEntry{}); err != nil {
panic(err)
}
row, err := w.Get([]byte("key"))
if err != nil {
panic(err)
}
fmt.Printf("Seq: %d Value: %+v\n", row.Seq, row.Value)
// flushed represents the immutable pages which were flushed to storage, but
// have not yet been confirmed as written to memtable.
flushed, err := w.Flush()
// NOTE: during this time any calls to Get([]byte("key")) will continue to return
// values for any RowEntries in the immutable pages.
it := flushed.Iterator()
for {
entry, err := it.NextEntry()
// Set these entries into the memtable
// ....
}
// Informs the immutable pages we are done with them.
flushed.Finalize() // or Close() or Confirmed()?
// "key" should not be in the WAL, as "flushed" has been finalized.
_, err := w.Get([]byte("key")) // No key found
// NOTE: If the flush to memtable fails for some reason, the next call to
// w.Flush() will include any immutable pages which have NOT been finalized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment