ZenGround0

Motivation

Today snapshotting of filecoin state into a CAR file from lotus node takes 2.5 hours on a 256 GB RAM machine. This is the bottleneck for proviing snapshots. The other phases are SHA256 hashing (10 minutes) S3 upload (20 minutes).

This snapshotting time is even in the presense of significant hardware resources -- 256 G RAM + 64 G Swap. Filecoin snapshots are smaller than this today. Latest snapshot shows ~120 G for snapshot size: ipfs://bafybeia5wm3jpf4nviu66w5sz3gu3xcmtwmsasswoar2bnupu3t5wdyl6y/

It should be possible to reduce lotus snapshotting time to the same order as other steps and remove the bottleneck. A nice feature of the proposed approach to do this is that it should interoperate well with forest snapshot generation and potentially speed that up significantly as well

Overview

ZenGround0

Summary

The FVM brings smart contracts to filecoin allowing developers to do productive novel things with verifiable data. One obvious thing to do is allow clients to put data on filecoin more easily. Enabling onboarding small pieces is better for users but worse for scale which motivates bridging to higher throughput lower latency lower cost systems like blockchain L2s. Some redesigning of the standard client contract flow is needed for things to work on L2s. Essentially the contract should be split into two halves: an expensive contract onboarding, aggregating and paying for data and an efficient contract transmitting proof of data storage.

FilOz is building a prototype of this here: https://github.com/ZenGround0/onramp-contracts/tree/main so that developers can build applications more easily following this design.

We're currently prototyping. First getting it working with L1 and L2 == Filecoin, then getting it working with true bridging on calibration net. Once we're done we'll measure performance

ZenGround0

Practical on chain on ramps

Zenground0

May 1 2024

An “on-ramp” is a system that helps users easily store data on filecoin. An on chain on ramp is such a system that works by sending blockchain messages to blockchain systems. With the FVM we are able to build on-chain on-ramps but they don’t exist yet. There is at least one fledgling project – Eastore, running on calibration net– that is fairly far along. This document lays out the issues and opportunities in quickly deploying smart contracts to filecoin and L2s that get us web3 native on-ramps in 3-6 months.

Disclaimer

FilOz is not interested in building or maintaining new production systems. This presents an extra challenge in figuring out how to coordinate the necessary work and system ownership. This document will ignore this altogether.

ZenGround0

Fil_Fil Product: 
a small incentivized data network overlay on top of filecoin specifically for filecoin snapshots
User requirement: snapshots are incentivized to be stored for a long time, ~2-5 years with replication
User requirement: snapshots are committed in real time < 1 hour after they are generated 
User nice-to-have: snapshots are available for content delivery on the saturn network

Big idea: providers participate in the network for an incentive.  Users of the network get guarantee that snapshot data is stored and (ideally) a reliable way to get the snapshot data out if they pay.  Basically it is like filecoin but for a tiny subset of specific data.

Users: 1. People installing lotus who need to sync (ideally FilecoinFoundation would subsiize saturn payments for people downloading from lotus-chain management website within reason 2. SPs who are out of sync 3. Devs / analysts who need historical data

ZenGround0

AMT Churn Explanation

Background

Market Churn

Churn is defined to be the size in bytes of the data in a datastructure that is overwritten over some period of time. Because GC in lotus is somewhat difficult lots of churn in filecoin state has a high system cost

We learned after nv19 that AMT churn is > 100x higher than optimal

ZenGround0

#!/bin/bash

# Use a trailing slash in directory if not provided
[[ "$2" != */ ]] && dir="$2"/ || dir="$2"

./lotus state --tipset=@$1 compute-state  --json | jq --argjson height "$1" '. + {Height: $height}'  >  "${dir}"gas-trace-"$1".json

cat "${dir}"gas-trace-$1.json  | jq -c '.Trace | .[]  | select( .Msg | .To== "f03") | .ExecutionTrace | .Subcalls | .[] | select( .Msg | .To == "f04") | .Subcalls | .[] | select( .Msg | .Method == 12) | {"To":( .Msg | .To) , "Gas": (reduce (.. | .GasCharges? | select(.) | .[])  as $gas ({tg: 0}; {tg: ($gas.tg + .tg)}))} ' | jq --argjson height "$1" -s '[.[] | . + {Height: $height}]'  > "${dir}"crongas-"$1".json

cat "${dir}"gas-trace-$1.json  | jq -c '.Trace | .[]  | select( .Msg | .To== "f03") | .ExecutionTrace | .Subcalls | .[] | select( .Msg | .To == "f04") | .Subcalls | .[] | select( .Msg | .Method == 12) | {"To":( .Msg | .To) , "Gas": (reduce (.. | .GasCharges? | select(.) | .[] | select(.Name == "wasm_exec") ) as $gas ({tt: 0}; {tt: ($gas.tt + .tt)}))} ' | jq --argjs

ZenGround0

Contract as a Deal Client for Archiving Filecoin Blockchain Message Data

Motivation

1. Use web3 infrastructure to solve the filecoin message archive problem currently solved with web2 clouds
2. Provide an EVM contract template for contract as a client data storage on the filecoin network
3. Identify sticking points in the current model of data preparation, data transfer and deal negotiation for contract as a client storage deals. For example learn how the ask protocol should change to support this model.
4. Demonstrate a new model for FIL+ auditing with significantly better fraud resilience properties

The core idea is that as system and protocol developers we will benefit from building a small application to understand how to adapt the network to similar applications. The nice thing is that fil infrastructure has a concrete application today which serves PLs broader goal of using our own tech.

ZenGround0

function devnet-boost
    set -x BOOST_EXEC_PATH /Users/zenground0/pl/repos/boost
    set -x LOTUS_EXEC_PATH /Users/zenground0/pl/repos/lotus
    cd $LOTUS_EXEC_PATH

    set -x FULLNODE_API_INFO (./lotus auth api-info --perm=admin | sed 's/^[^=]*=//')
    set -x MINER_API_INFO (./lotus-miner auth api-info --perm=admin | sed 's/^[^=]*=//')
    set -x PSD_WALLET (./lotus wallet new)
    echo "Sending 100 FIL to $PSD_WALLET"
    ./lotus state wait-msg (./lotus send $PSD_WALLET 100)

ZenGround0

ZenGround0

addr,power_summary,power_percentage,debt,locked_funds,debt_minus_locked_funds,debt_minus_locked_funds_in_rounded_FIL
f01155,(454.9 TiB / 489.9 PiB),0.0906%,738165819535979446420,163461236877981722109,574704582657997724311,575FIL
f015902,(1.312 TiB / 489.9 PiB),0.0002%,15892725699167755436,0,15892725699167755436,16FIL
f08385,(288 GiB / 489.9 PiB),0.0000%,1124620050307575561,0,1124620050307575561,1FIL
f02419,(2.312 TiB / 489.9 PiB),0.0004%,178541895364964059964,0,178541895364964059964,179FIL
f022970,(384 GiB / 489.9 PiB),0.0000%,157040318843098910,0,157040318843098910,0FIL
f015799,(32 GiB / 489.9 PiB),0.0000%,15771041290578916234,0,15771041290578916234,16FIL
f08399,(1.156 TiB / 489.9 PiB),0.0002%,4363625856165831805,0,4363625856165831805,4FIL
f015805,(1.344 TiB / 489.9 PiB),0.0002%,16892993524550778769,0,16892993524550778769,17FIL
f022791,(384 GiB / 489.9 PiB),0.0000%,2328995567473538653,0,2328995567473538653,2FIL