Skip to content

Instantly share code, notes, and snippets.

@xlab
Last active April 4, 2022 17:21
Show Gist options
  • Save xlab/6e204ef96b4433a697b3 to your computer and use it in GitHub Desktop.
Save xlab/6e204ef96b4433a697b3 to your computer and use it in GitHub Desktop.
Golang split byte slice in chunks sized by limit
func split(buf []byte, lim int) [][]byte {
var chunk []byte
chunks := make([][]byte, 0, len(buf)/lim+1)
for len(buf) >= lim {
chunk, buf = buf[:lim], buf[lim:]
chunks = append(chunks, chunk)
}
if len(buf) > 0 {
chunks = append(chunks, buf[:len(buf)])
}
return chunks
}
const PackSizeLimit = 5 * 1024 * 1024
// [a b c] -> [[a, b], [c]]
// where (len(a)+len(b) < MAX) and (len(c) < MAX),
// where (len(a) < MAX) and (len(b) < MAX) and (len(c) < MAX),
// but (len(a) + len(b) + len(c)) > MAX
func splitForJoin(chunks [][]byte, lim int) [][][]byte {
var result [][][]byte
for len(chunks) > 0 {
for i, v := range chunks {
if size+len(v) > PackSizeLimit {
chunks = chunks[i:]
break
}
}
// TODO
}
return result
}
@JulianKnodt
Copy link

Just curious on line 9, in
chunks = append(chunks, buf[:len(buf)])
why do you have to do buf[:len(buf)], because that just captures a copy of that part of the slice, but wouldn't just buf suffice?

@pablodz
Copy link

pablodz commented Dec 15, 2021

Great work

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment