Skip to content

Instantly share code, notes, and snippets.

@meiqimichelle

meiqimichelle/ipfs-pinning-current_state.md

Created November 8, 2019 11:02
Show Gist options
  • Save meiqimichelle/25d2221c08807841be7764b22f4a5bb5 to your computer and use it in GitHub Desktop.
Save meiqimichelle/25d2221c08807841be7764b22f4a5bb5 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
ipfs-pinning-current_state.md

Background / current state / pain points: pinning on IPFS

What is a pin?

In IPFS, a “pin” is a DAG (for recursive-pins, as identified by its root CID) or a block (for direct-pins), that is not to be removed during Garbage Collection (GC). This “saves” the pinned information so that it is available for the user in the future, or available for others to discover and perhaps view or “save/pin” themselves. The pinned information is discoverable to the extent that the peer itself is accessible on the network (ie, if pinned to an IPFS network running privately, then it is only available within that private network; by default, an object pinned on a single peer is available on the public IPFS network).

From a semantics perspective, and to paraphrase @whyrusleeping from this note, IPFS tries to make it feel as though all objects are local. There is no “retrieve this file for me from a remote server;” the commands all act the same way no matter where an object is located. However, users want to be able to control what they keep. Pinning is the mechanism that allows a user to tell IPFS to always keep a given object local and accessible. IPFS caches objects locally for a short time after the user performs any IPFS operation on them, but these objects may get garbage collected eventually. To prevent this, users can “pin” the hash they care about. Objects added through ipfs add are pinned recursively by default because IPFS assumes that if you go to the trouble of adding an object to the network, you care about it enough to want to keep it around.

Types of pinning

Pinning process

  • Sync: the user needs to wait until it is pinned (ipfs).
    • 😕Users must wait until sync pinning is 100% complete until they can do another action or end a process, and the objects being pinned must continue to be accessible throughout (aka, don’t close your laptop)
  • Async: the user submits the pin (Cluster, pinning services) and the service pins in the background
    • 🎉Intermediary services are providing a layer of tooling that makes it easier for people to engage with “saving” objects on the dweb.
    • 😕Even for these intermediary services, communicating that it takes some amount of time between when you “save” or “add” something to IPFS, and when it’s actually available on IPFS, is a challenge.

Pin types

  • Recursive (pins a given block and all of its children)
    • This is the type of pinning most end users and developers are concerned with.
  • Direct (pins a single block)
  • Indirect (what you call a block that's been pinned recursively; a child of a recursively-pinned block)

Types of pinning interactions

Pinning to a single peer (ipfs)

  • Commands (CLI)
    • ipfs add
      • “I have an object on my machine, and I want to put it on IPFS.” This is the idea that most end users and developers associate with adding.
      • Can only take raw bytes.
      • 🎉By default, this command pins after adding. IPFS assumes that if a user goes to the trouble of adding an object to IPFS, they also want to keep it around. This meets end user and developer expectations.
    • ipfs pin add
      • Basic: “I already have an object in my datastore, and I want to keep it around by “saving” it.”
        • Someone sends you a quokka picture over IPFS.
          • “Hey I think you will like this picture”
          • “Sure let me have a look” (you look at the photo)
        • That’s a great quokka, and you want to keep it so that it doesn’t get GC’d/cleared from cache/cleared from datastore. It’s in your datastore already because you viewed it over IPFS.
          • ipfs pin add
      • Advanced dweb: “I already have an object in my datastore, and I want to help make it available on the network by co-hosting it.”
        • You are browsing Wikipedia via IPFS, and you want to make sure some very important quokka information is always available.
        • Since you are browsing via Companion with a local IPFS node, once you’ve viewed that Wikipedia page, its objects are already in your datastore. You want to keep them so they don’t get GC’d, and are available via your node.
          • ipfs pin add
      • Can only take a hash.
    • ipfs add vs ipfs pin add
      • ipfs add will take a ‘raw’ file and chunk and add it to the datastore, whereas ipfs pin add needs the file to exist in the network, and will actually ask the network for it if it doesn’t already exist in the local datastore.
      • 😕These commands are confusing in part because although they both use the word “add,” these are not the same “add.” Ideally, these actions would have been called something more clearly different in our API.
        • ipfs add = chunk my file into my datastore/network (add), and don’t delete them (the pin that happens automatically after add)
        • ipfs pin add = don’t GC these existing datastore/network blocks (pin -- and what’s that add for? Shrug. Certainly not file chunking, as in ipfs add.)
      • 🎉Fortunately, we can use different terms for end users, and hide this complexity. If we could, it would be even better to rename ipfs pin add to just ipfs pin or another solution that doesn’t involve the word add.
    • ipfs pin rm
      • IPFS does not remove blocks. It justs removes a given from the list of recursive pins.
      • In order to remove the unpinned blocks from IPFS (so that they will not be advertised or provided to the network), IPFS garbage collection (ipfs repo gc) needs to run. This checks the list of pins, and traverses recursively for every pin. It puts all the blocks that are indirectly referenced from the root pins in a bag, and then removes blocks that are not in that bag.
      • 😕This is a very expensive procedure, and while this happens nothing else can happen; the peer becomes unusable.
      • 😕There isn’t a way for a node to express “I'm not providing that anymore” on the DHT. Records expire after 12 hours, but even if the peer has GC'ed it may still be contacted for blocks it doesn't have until that happens.
  • APIs
    • HTTP API: POST /v0/api/pin/add, /v0/api/add
    • API Bindings: Pin/Add methods in Go and JS that trigger HTTP API requests
    • Core API: Pin method can be used to pin when programmatically running an IPFS node

Single peer + GUI

  • IPFS WebUI
    • IPFS WebUI is a browser-based interface for a user’s IPFS node. Via this interface, users can check on node stats, explore the IPLD-powered merkle forest, see peers around the world, and manage their files, without needing to touch the CLI.
    • It is a React app that communicates with a user’s local node via ipfs-http-client.
    • Regarding pinning:
      • WebUI is both a view into what is pinned and what files/folders you have in your MFS (your local datastore in general can have blocks that are not displayed on the Files screen), and a place where you can take actions and have those reflected across IPFS.
      • 🎉You can graphically “+Add”
        • File
        • Folder
        • From IPFS
        • New Folder
      • Once a file or folder is in your Files interface, you can:
        • View the file or folder.
        • Delete
          • 😕See ipfs pin rm above for UX issues with ‘delete’
        • Rename
          • 🎉Works as advertised.
        • Download
          • 🎉Works as advertised.
        • Inspect
          • 😕Opens file or folder in “Inspect” page. It is not very clear what this does, or what it’s for, when coming from the Files page.
        • Copy hash
          • 😕Works as expected and copies QmHash to clipboard, but there’s no UI indicator that it’s been successful, as with other interactions. This makes the user unsure if the action has ‘worked.’
        • Share
          • 😕Opens a modal for a user to copy a hash that’s prepended with an http path to the item. This does not meet user expectations because Web 2.0 “Share” icons typically provide several direct-share-to-app options plus something called “copy link” or similar. The WebUI “Share” is actually == “copy link,” and users are being asked to “link” to something that isn’t usually linked to in today’s Share systems (ie, one would typically “send” or “share” a doc or image directly with someone, not send them a link to the item; links are for webpages and similar). There is an opportunity to improve this interaction quite a lot, as even in file systems such as Dropbox, linking and sharing isn’t always straightforward.
        • Pin or Unpin
          • 😕It is not clear how the ‘pin’ icon and options on the “files” part of the Files Page relate to the pins on the “pins” part of the Files Page. The numbers of items differ. Also, the files in “files” are not GC’d, so why don’t they all have pin icons? What does the pin icon mean here?
        • 😕There are different options depending on whether the user selects the drop-down, or selects the checkbox next to the item.
      • In the Pins interface, you can do a subset of the actions above on the pin directly:
        • Share
        • Copy hash
        • Download
        • Inspect
        • Unpin
        • 😕These have the same positive/negative notes as noted above in ‘files’, with the additional confusion of -- how can I, as a user, decide which pin I want to act on? All that’s listed are hashes with no other identifying information, and in fact, those hashes are listed twice (as item title, and in the slot where the hash would go if there were a proper title for that item).
      • 😕There’s a confusing dichotomy between the “files” and “pins” on the File Page. Right now, “files” in WebUI shows only files and directories in MFS, and MFS is a subset of everything in local datastore:
        • ipfs add adds data to the local local datastore (repo), but it does not add it to MFS. To add a file or directory to MFS via CLI, you need to:
          • ipfs add to get CID
          • Add the CID to MFS via ipfs files cp /ipfs/{CID} /name-on-mfs
        • When the user adds a file or a directory via WebUI, it does both steps (ipfs add + ipfs files cp to MFS)
      • ...while “pins” shows objects that the user has “pinned” (ipfs pin add), even though both are technically “pins” behind the scenes, and the user expects to see both via the same interface (ie, I want to see what I’ve got in my local repo).
        • 😕Confusion points for users: Why can I see the names of files, but not names of pins? Why can I take certain actions on files/folders, but not on pins? Why, when I inspect files versus pins, or when I inspect different files, do I see different things? Some have more file-looking items there with names, some have one big dot, some have lists of hashes without real titles associated with them. There’s a confusing lack of intention around what is shown as a “file,” a “pin,” and what the interface on the “Inspect” page means (which of what is shown is a “file” or a “pin” or something else?).
      • 🎉The GUI team is aware of many of these painpoints, and has existing plans to improve this interface that will be informed by this PRD
  • IPFS Desktop
    • IPFS Desktop allows users to run an IPFS node on their machines without having to bother with command line tools. With it, they have the power of the IPFS WebUI, plus a handful of helpful shortcuts. It is available on Mac, Windows, and Linux.
    • IPFS Desktop's main feature is to allow you to have the IPFS daemon always running in the background, and it provides auto-update mechanism, which makes sure user is running the latest version of IPFS daemon behind the scenes.
    • Regarding pinning:
      • On Windows, you can right click on files and folders to add them to IPFS.
      • On macOS, you can drag and drop objects to the tray icon to add them to IPFS.
      • To view added files, the user has the same experience as via the WebUI (see above).
  • IPFS Companion
    • IPFS Companion is a browser extension that enables everyone to access IPFS objects the way they were meant to be accessed: from a locally-running IPFS node. It is available on Firefox, Firefox for Android, Chrome, Brave, Opera, and Edge.
    • Companion detects requests on websites for IPFS-like paths (/ipfs/{cid} or /ipns/{peerid_or_host-with-dnslink}), and redirects and loads them from a local gateway:
    • Companion will also detect the presence of DNSLink in DNS records of visited websites and redirect HTTP requests to a local gateway (the browser will load the website from IPFS).
    • Companion provides additional actions for pages loaded from IPFS:
      • Pin/Unpin of IPFS resources (via API)
      • Copy canonical IPFS address
      • Copy shareable URL to resource at preferred public gateway
    • Regarding pinning:
      • Has "Share files via IPFS" in its browser extension pop-up menu. Clicking on this opens a new window where you can upload a file or directory. Doing so adds that item to IPFS (and pins by default unless a checkbox is unchecked).
        • 🎉There is a plan to change this behavior and add files to MFS instead, so the user can see shared files in WebUI's Files screen.
      • Adds the ability to right click, and “Add selected text to IPFS.” This opens the selected text in a new browser window with its QmHash+http-accessible-url in the address bar. This new pin is then listed in the the user’s WebUI/Desktop “Files Page/pins”.
        • 😕What is the user need behind selecting text and adding it to IPFS?
  • IPFS GUI apps <> public IPFS network <> expectations of privacy
    • Our GUI apps, by default, only interact with the public IPFS network. All pins and files displayed via their interfaces are technically findable by other IPFS nodes. (One can point an IPFS node to a private network, if desired, and Companion is just a proxy to IPFS nodes.)
    • 😕In a distributed system in which other people can “save”/pin information, sometimes data is neither available in the way folks have gotten used to on Web 2.0, and nor is it “deleted” in the way folks understand that concept today.
    • 😕There is a mismatch between technical network privacy and what end users expect “local” and “privacy” to mean. This is a UX hurdle that should be treated with care as we move forward, as people expect their local machine to automatically be “private” in the way they understand “this is my house versus that is your house, and I need to invite you in before you can see what's inside,” unless this is clearly messaged otherwise (ex dropbox / google drive folders).

Pinning to multiple peers (Cluster)

  • Commands (CLI)
    • ipfs-cluster-ctl add
      • 🎉By default, this command pins after adding.
      • The ipfs-cluster-ctl add command is very similar to ipfs add except that the ipfs add command only adds to a local IPFS peer, and ipfs-cluster-ctl add adds to several Cluster peers at the same time. How many it adds to depends on the replication factors the user sets as command flags or the defaults in the configuration file.
      • 🎉Cluster considers a pin add operation to be successful when the cluster-pinning stage is finished. This means the pin has been ingested by Cluster and that things are underway to tell IPFS to pin the content. If IPFS fails to pin the content, Cluster will know, report about it and try to handle the situation. It works this way because cluster-pinning stage is relatively fast but the ipfs-pinning stage can take days. Therefore, the second stage happens asynchronously once the cluster-pinning stage is completed.
    • ipfs-cluster-ctl pin add
      • Adds content from the IPFS network to a Cluster. The ipfs-cluster-ctl pin add operation is similar to the ipfs pin add one, but allows the user to set Cluster-specific flags, such replication factors or the name associated with a pin.
    • ipfs-cluster-ctl pin rm
      • Pins can be removed from a Cluster at any time. They are then treated by IPFS as any other ipfs pin rm command.
  • Cluster APIs
    • HTTP
      • Rest API: POST /pins/
      • Proxy API: POST /v0/api/pin/add, /v0/api/add
    • API Bindings: Pin/Add methods in Go and JS that trigger Rest API requests
    • Go API: Pin method can be used to pin when programmatically running an IPFS Cluster peer
  • Types of Clusters
    • Standard Cluster: all peers have equal “status;” they all control pinning/unpinning. All data on a Standard Cluster replicates across all peers.
      • 🎉Cluster provides a layer of tooling that makes IPFS / availability on the dweb easier to understand and develop on because it solves for a very common need (automated availability) that developers or end users would, first, have to understand is missing from single-peer IPFS usage (no small feat for those new to the space), and second, otherwise have to build themselves.
    • Collaborative Cluster: some peers have less “status;” they are followers that can contribute storage, but cannot control what gets pinned/unpinned. Full peers in a Collaborative Cluster control what gets pinned/unpinned, and their decisions replicate across all peers.
      • 🎉Users have requested this feature from IPFS, and we already have it!
      • 😕But they’re requesting it because they don’t know that we already have it.

Third-party pinning services

  • Pinning services (such as Pinata, Infura, and 3box) offer APIs and browser interactions that act like Amazon S3 and similar Web 2.0 data store services.
    • 🎉Services like these allow developers to build on the dweb without requiring them to replace their entire stack/mental model of development in one go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment