Skip to content

Instantly share code, notes, and snippets.

View joeltg's full-sized avatar

Joel Gustafson joeltg

219 followers · 10 following
View GitHub Profile
@joeltg
joeltg / notes-roadmap.md
Last active December 6, 2018 06:26
Technical Notes and Roadmap for the Underlay

Technical Notes and Roadmap

Data model

Split into two stages to allow IPLD to develop further:

  1. Now: JSON-LD file-based model. JSON-LD is the future of RDF and it's already popular. The simplest, most future-friendly and semantically-idiomatic model to get the Underlay started now is to publish JSON-LD files that:
  • are flattened relative to one top-level context,
  • address every node with either absolute or fragment-relative URIs (no blank or path-relative nodes),
  • link to other nodes with the IPFS URI scheme dweb:/ipfs/Qm..., using fragments for nodes within documents: dweb:/ipfs/Qm...#node-id
  • use the PROV vocabulary with named graphs to capture "first-class" provenance as JSON-LD itself
  • are persisted to IPFS with the files API, and
@joeltg
joeltg / interfaces.md
Last active December 6, 2018 06:26

Underlay Interfaces

Every interface implementation is a class with at least two static properties type: string[] and data: {[property: string]: string}, and (depending on the interface(s) it implements) zero or more public async functions. Every interface class will be constructed with new Class(options: {}).

Namespace

Indicates a service that manages some context-scoped concept of identity

  • type: ["http://underlay.mit.edu/interface/namespace"]
  • data.namespace: string
    • a static property containing the URI of the namespace indexed by the resolver
    • "http://www.wikidata.org/"
@joeltg
joeltg / setup.sh
Created October 19, 2018 12:49
Setup for IPFS Cluster Node (Linode w/ Debian 9)
apt update
apt upgrade -y
apt install -y make git jq
wget https://dl.google.com/go/go1.11.1.linux-amd64.tar.gz
tar -C /usr/local -xzf go1.11.1.linux-amd64.tar.gz
rm go1.11.1.linux-amd64.tar.gz
echo "export PATH=\$PATH:/usr/local/go/bin:/root/go/bin" >> /etc/profile
source /etc/profile
go get -u github.com/ipfs/ipfs-update
ipfs-update install latest
@joeltg
joeltg / islands.md
Last active December 6, 2018 06:25

Schema translations

I'm going to use the words "schema" and "ontology" interchangeably and there's nothing you can do to stop me.

There's lots of structured data out in the wild, but it's only structured internally, relative to itself. For example, IMDB will let you download CSVs of its archives, but they're in IMDB's own schema, with made-up IMDB names for all the columns. If you're consuming their data, you still need to read their interface docs and transform the data into an externally compatible format. Their data is only usable intentionally.

What we'd really love is for data to also be usable incidentally - reachable, retrievable, and interpretable from outside worlds without any human in the loop. One way to achieve this is through standardization (getting IMDB to switch all their property names to fully-qualified URIs in some standardized movie-ish ontology), but a far more practical & scalable strategy is through self-des

@joeltg
joeltg / cluster.sh
Last active November 5, 2024 22:48
IPFS Cluster Setup
#!/bin/bash
set -x
USER=ec2-user
GROUP=ec2-user
# Update these when appropriate
IPFS_VERSION=v0.4.21
IPFS_CLUSTER_VERSION=v0.10.1
@joeltg
joeltg / loader.go
Last active June 6, 2019 17:17
package loader
import (
"net/url"
"strings"
cid "github.com/ipfs/go-cid"
ipfs "github.com/ipfs/go-ipfs-api"
ld "github.com/piprate/json-gold/ld"
)
@joeltg
joeltg / tika-provenance.json
Last active January 8, 2019 14:41
Schema.org markup for Tika Server and its endpoints
{
"@context": {
"prov": "http://www.w3.org/ns/prov#",
"format": "http://www.w3.org/ns/formats/",
"@vocab": "http://schema.org/"
},
"@graph": [
{
"@id": "dweb:/ipfs/QmScWKwDmJP9nVou2jVVCtRLQQNcWBMXwoJnoa4RULL8wn#_:c14n74",
"url": "https://tika.apache.org",
@joeltg
joeltg / tika-context.json
Last active April 15, 2019 05:10
Namespace prefix mappings for the RDF properties produced by Apache Tika, for use as a JSON-LD context
{
"dc": "http://purl.org/dc/elements/1.1/",
"dcterms": "http://purl.org/dc/terms/",
"Iptc4xmpCore": "http://iptc.org/std/Iptc4xmpCore/1.0/xmlns/",
"Iptc4xmpExt": "http://iptc.org/std/Iptc4xmpExt/2008-02-29/",
"plus": "http://ns.useplus.org/ldf/xmp/1.0/",
"cp": "http://schemas.openxmlformats.org/package/2006/metadata/core-properties/",
"extended-properties": "http://schemas.openxmlformats.org/officeDocument/2006/extended-properties/",
"photoshop": "http://ns.adobe.com/photoshop/1.0/",
"xmp": "http://ns.adobe.com/xap/1.0/",
@joeltg
joeltg / elastic.json
Created April 15, 2019 13:35
ElasticSearch config for the Prior Art Archive
{
"mappings": {
"doc": {
"properties": {
"text": { "type": "text" },
"title": { "type": "text" },
"fileUrl": { "type": "keyword" },
"organizationId": { "type": "keyword" },
"uploadDate": { "type": "date" },
"publicationDate": { "type": "date" },
@joeltg
joeltg / jsonld.d.ts
Last active April 17, 2019 17:37
TypeScript bindings for JSON-LD
type Container<T> = T | T[]
type RecursiveContainer<T> = T | RecursiveArray<T>
interface RecursiveArray<T> extends Array<RecursiveContainer<T>> {}
declare namespace JsonLd {
export type Graph = Container<Node>
export type Node = {
"@id"?: string
"@type"?: Container<string>