filbeam-as-proxy.md

FilBeam as Proxy

Hypothesis

SPs don't want the FilBeam cache sitting in front of them, as it means that SPs will not be rewarded for every retrieval. Instead, they are only rewarded for cache misses (a percentage of retrievals).

Proposal

Disable the Filbeam cache, and proxy every request through to SPs. This is what is going to be explored in the rest of the document.

Alternative considered

Keep the proxy, as it is beneficial for content requesters, leading to higher availability. However, also reward SPs on cache hit.

The problem with this approach is that this will lead to self dealing, as SPs are now rewarded for not doing any work, and anyone can create FilBeam requests.

Context

This is what will be removed from the current retrieval flow:

sequenceDiagram
  participant c@{ "type": "entity" } as Client
  participant wr as Worker: retriever
  participant di@{ "type": "database" } as D1: Index
  participant dm@{ "type": "database" } as D1: Metrics
  participant sp as SP
  c ->> wr: Request {clientAddress,commP}
  wr ->> di: Query index
  rect rgb(255, 0, 0)
    wr ->> wr: Check cache
    alt is not in cache
      rect rgb(255, 255, 255)
        wr ->> sp: Retrieve file
      end
      wr ->> wr: Populate cache
    end
  end
  wr ->> c: Serve file
  wr ->> dm: Write metrics

Loading

Removed features

• Caching
• Traffic spike protection: Without a cache in front, SPs are expected to be available for every incoming request.

New features

As a side effect, this unblocks HTTP range requests, which are currently not supported by Cloudflare's cache.

What values does a Filbeam proxy still serve?

Given we remove the cache, a critical part of CDN infrastructure, it's fair to ask what value FilBeam still serves. The focus will shift from CDN features towards a retrieval incentive layer. However, some of the availability features are still preserved.

• Paid retrievals: SPs are rewarded for every byte served
• One piece retrieval URL for multiple SPs: Any SP serving the piece will be attempted for retrieval, failures will be hidden from the client.

Potential in-between feature:

• Cache read on unavailability: In the case no SP can serve the retrieval, it is possible that only then FilBeam will attempt to serve the retrieval from its cache. In that case however one would ask, why doesn't it always try to do that. It would need to maintain a healthy cache, but only use it when SPs fail.

Simplifications

• Pricing: Currently, one has to estimate how many bytes of retrieval $1 buys, as it is distributed between cache-hit and cache-miss traffic, on an unknown ratio. With this proposal, one can now say exactly how much one byte of traffic will cost.
• Architecture: No cache to maintain
  • This also makes it easier to eventually move away from Cloudflare, as a FilBeam node now just needs to be a smart proxy with a database, and not contain any piece cache.
• Cache-misses will be slighty simpler and therefore slightly faster.

Non-simplifications

• For now there will still need to be 3 payment rails per data set, as requests still both reward the SP and FilBeam, and a payment rail can only have one recipient
  • This can be simplified in the future, with the recipient being a forwarder contract, that shares funds between SP and FilBeam on a fixed ratio

What this means for FOC as a product

• Iff SPs are less reliable than Cloudflare, there will be lower availability of data