FilBeam architecture

filbeam-architecture.md

FilBeam

Sites

• filbeam.com product
• docs.filbeam.com docs
• dashboard.filbeam.com dashboard
• filbeam.io cdn endpoint

Repositories

• filbeam/worker workers, subgraphs
• filbeam/contracts smart contracts
• filbeam/telegraf monitoring

Stack

The stack mainly consists of Cloudflare (workers, D1) & Goldsky. Auxiliary components are Fly.io for other deployments, InfluxData, Solarwinds & Grafana.

Index

Populate an off-chain representation of pieces, data sets & deals/funds, for fast retrieval.

flowchart LR
Contracts --> Subgraph
Subgraph --> Webhooks
Webhooks --> wi[Worker: indexer]
wi --> di[D1: Index]

Loading

Contracts & Events

• FWSS
  • DataSetCreated
  • PieceAdded
  • ServiceTerminated
  • CdnServiceTerminated
  • CdnPaymentRailsToppedUp
• PDPVerifier
  • PiecesRemoved
• FilBeamOperator
  • CDNSettlement

Retrieval

Fast, cached and metered serving of pieces to dumb clients via HTTPS.

https://{clientAddress}.filbeam.io/{commP}
https://{clientAddress}.calibration.filbeam.io/{commP}

sequenceDiagram
  participant c@{ "type": "entity" } as Client
  participant wr as Worker: retriever
  participant di@{ "type": "database" } as D1: Index
  participant dm@{ "type": "database" } as D1: Metrics
  participant sp as SP
  c ->> wr: Request {clientAddress,commP}
  wr ->> di: Query index
  alt is not in cache
    wr ->> sp: Retrieve file
    wr ->> wr: Populate cache
  end
  wr ->> c: Serve file
  wr ->> dm: Write metrics

Loading

See also

• https://docs.filbeam.com/how-it-works/retrieval-and-fallback

Note: SPs are trusted to serve correct piece content, the CID isn't verified. This is because even with the fastest method available (Hugo's commp library), the overhead would still be too much for web2 performance needs, and frequently lead to worker timeouts. To turn this on, one needs to go back to the drawing board.

Payments & settlements

FilBeamOperator

https://github.com/filbeam/contracts/blob/6ef1b43f3abe78f9fb525af773d7db1ca68f2ec0/src/FilBeamOperator.sol

Not upgradeable, but operator address can be changed in FWSS

FilbeamOperator implements one-time payments to account for egress-based quota depletion.

Each data set that includes FilBeam service, will have 2 extra active payment rails, in addition to the pdp one:

• cacheMissRail (pays SP for egress to FilBeam)
• egressRail (pays FilBeam for egress to the world)

sequenceDiagram
  participant wr as Worker: retriever
  participant d@{ "type": "database" } as D1: Metrics
  participant wu as Worker: reporter
  participant c as Contract: FilBeamOperator
  participant ws as Worker: settler
  participant fwss as Contract: FWSS
  loop after request
    wr ->> d: Update quotas
  end
  loop periodic
    wu ->> d: Query usage
    wu ->> c: recordUsageRollups()
    c ->> c: Update dataSetUsage
  end
  loop periodic
    ws ->> c: settlePaymentRails()
    c ->> c: Read dataSetUsage
    c ->> fwss: settleFilBeamPaymentRails()
  end

Loading

See also:

• https://docs.filbeam.com/how-it-works/quota-system
• https://docs.filbeam.com/how-it-works/payment-model
• https://docs.filbeam.com/how-it-works/usage-reporting

IPFS support

Work in progress: Serve IPFS retreivals #312.

https://1-{base32(dataSetId)}-{base32(pieceId)}.ipfs.filbeam.io/{filePath}

# redirects to the above
https://ipfs.filbeam.io/{WalletAddres}/{IpfsRootCid}/{filePath} 

TODO

• Productionize
• Refactor code shared by piece-retriever and ipfs-retriever
• Match IPFS trustless GW options
• Add metering

Security

In order not to violate laws, CIDs are periodically scanned against badbits, and wallet addresses against chainalysis.