Skip to content

Instantly share code, notes, and snippets.

View rvagg's full-sized avatar

Rod Vagg rvagg

View GitHub Profile
@rvagg
rvagg / main.go
Last active September 6, 2024 06:46
podsi piece parsing to match boost
package main
// attempt to replicate https://github.com/filecoin-project/boost/blob/main/piecedirectory/piecedirectory.go behaviour
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
@rvagg
rvagg / inspectblock.go
Created August 2, 2022 00:50
Inspect a dag-pb IPLD block and traverse the immediate links using various techniques to compare for consistency
package main
import (
"bytes"
"context"
"fmt"
"io"
"os"
"github.com/ipfs/go-cid"
{"cid":"bafybeihux5wmr7tia4eimuodo5hapw6cxl2qbjfnj4nd6lugrw5jylbjsy","event":"connect","eventDetails":null,"eventTime":"2022-07-20T12:17:13.878442835Z","instanceId":"autoretrieve-unnamed","phase":"query","phaseStartTime":"2022-07-20T12:17:12.967932646Z","retrievalId":"065842cf-1b75-435b-8396-9a11c5d3b283","storageProviderId":"12D3KooWRnVTEA1jScyYsCwBxi1hYvxvoDDfPyLkow3eyX2B92c3"}
{"cid":"bafkreicpjzh2fqcyaws4cqysuary46skccfdv3f63d3rdwpy46ob4bsbk4","event":"connect","eventDetails":null,"eventTime":"2022-07-20T12:17:14.098088619Z","instanceId":"autoretrieve-unnamed","phase":"query","phaseStartTime":"2022-07-20T12:17:14.097663068Z","retrievalId":"37903ab7-a8f4-4398-837d-601ec57e30f3","storageProviderId":"12D3KooWRnVTEA1jScyYsCwBxi1hYvxvoDDfPyLkow3eyX2B92c3"}
{"cid":"bafybeiaphugi3dulcnxchx6jm7gx57z4nnkdht32dxagurxo4dulmxwqo4","event":"connect","eventDetails":null,"eventTime":"2022-07-20T12:17:14.315414746Z","instanceId":"autoretrieve-unnamed","phase":"query","phaseStartTime":"2022-07-20T12:17:13.518404076Z","ret
@rvagg
rvagg / multicodec_pull_203_discuss.md
Created August 25, 2021 05:39
Limitations of CIDs and IPLD codecs wrt various signalling problems

This post builds on the discussion multiformats/multicodec#203 and is posted here because it might be a bit too long and in-the-weeds and I doubt many will actually read this all anyway! But I'd like this as a record of an ongiong broader disucssion about these topics.


Working through the process of mapping blockchain block formats to IPLD has made me see this question slightly differently. First it was getting the full bitcoin chain format working, including the awkward segwit hacks, and now it's working with @i-norden through getting the full ethereum format mapped to IPLD (e.g.).

The primary goal we're trying to achieve with IPLD codec codes here (via CIDs usually) is describing what glasses we put on to see the data. We want to get something into the data model in memory from the raw binary we've been handed.

An obvious example of having different glasses would be these CIDs:

82 # array(2)
9f # array(Infinity)
82 # array(2)
52 # bytes(18)
2f6261636b757064732f6c6f672f68656164 # "/backupds/log/head"
58 43 # bytes(67)
313632373939333934302e6c6f672e63626f723b # "1627993940.log.cbor;"
61613539656265302d613938382d343938302d62 # "aa59ebe0-a988-4980-b"
6366332d3739613134346566623962313b313632 # "cf3-79a144efb9b1;162"
37393933393430 # "7993940"
@rvagg
rvagg / thought-bubble.md
Created September 10, 2020 05:11
High-level JS abstractions for navigating IPLD data (thought-bubble)

Continuing on from multiformats/js-multiformats#35 (comment), because I don't want to derail that but I'd like to seed more thinking about how we can move toward better high-level IPLD abstractions.

There are currently a few different visions about this topic, but mostly I think it's pretty grey because we need to experiment with it more to figure out what actually makes sense. I think some of our current core differences are around what to do with the block boundary. I'd like us to try and erase the block boundary more at the user-facing end (not entirely, it's an abstraction that has to leak to some degree because there are costs to pretending it doesn't exist). go-ipld-prime is pushing forward to a model that I think we can somewhat mirror in JavaScript, with some major differences - primarily in that we have the async boundary to deal with, and we can use plain JavaScript objects to represent complex things (they have to either pre-define their object shapes or us

{
"logFormatVersion": 1,
"jobId": "f7c77a9c-472c-4a0e-b80b-b776c42e1fad",
"status": "Accepted",
"statusSummary": "Ready for distribution",
"statusCode": 0,
"archiveFilename": "node-v14.0.0-nightly20200122c68fa207d7.pkg",
"uploadDate": "2020-01-22T04:18:58Z",
"sha256": "da3f7b4293a98afa17753a376e9b8240940b859eef305cdeeb874a1094097e78",
"ticketContents": [
@rvagg
rvagg / dpkg-l.diff
Created October 2, 2019 02:48
test-rackspace-debian8-x64-{1,2} dpkg -l differences
27c27
< ii ca-certificates 20170717~14.04.1 all Common CA certificates
---
> ii ca-certificates 20141019+deb8u4 all Common CA certificates
29,30d28
< ii cloud-guest-utils 0.29-1~bpo8+1 all cloud guest utilities
< ii cloud-image-utils 0.29-1~bpo8+1 all cloud image management utilities
32,34c30,32
< ii cloud-initramfs-dyn-netconf 0.25ubuntu1.14.04.2 all write a network interface file in /run for BOOTIF
< ii cloud-initramfs-growroot 0.25ubuntu1.14.04.2 all automatically resize the root partition on first boot
@rvagg
rvagg / arch-vers-perday.csv
Created May 5, 2018 00:19
Node arch+version download stats 2018
We can make this file beautiful and searchable if this error is corrected: It looks like row 3 should actually have 97 columns, instead of 36 in line 2.
arch,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv6l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,armv7l,x64,x64,x64,x64,x64,x64,x64,x64,x64,x64,x64,x64,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,ppc64le,arm64,arm64,arm64,arm64,arm64,arm64,arm64,arm64,arm64,arm64,arm64,arm64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,ppc64,x86,x86,x86,x86,x86,x86,x86,x86,x86,x86,x86,x86,s390x,s390x,s390x,s390x,s390x,s390x,s390x,s390x,s390x,s390x,s390x,s390x
version,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10,0.10,0.11,0.12,0.8,0.9,4,5,6,7,8,9,10
2018-01-01,0,0,0,0,0,129,15,210,35,190,99,0,0,0,0,0,0,83,18,149,98,266,193,0,1814,60,2380,3254,2,27718,4600,53654,16847,56933,17064