ZenGround0 · August 30, 2023 01:22
diff --git a/FIL__FIL.txt b/FIL__FIL.txt
 Fil_Fil Product: 
 a small incentivized data network overlay on top of filecoin specifically for filecoin snapshots
 User requirement: snapshots are incentivized to be stored for a long time, ~2-5 years with replication
 User requirement: snapshots are committed in real time < 1 hour after they are generated 
 User nice-to-have: snapshots are available for content delivery on the saturn network

 Big idea: providers participate in the network for an incentive.  Users of the network get guarantee that snapshot data is stored and (ideally) a reliable way to get the snapshot data out if they pay.  Basically it is like filecoin but for a tiny subset of specific data.

 Users: 1. People installing lotus who need to sync (ideally FilecoinFoundation would subsiize saturn payments for people downloading from lotus-chain management website within reason 2. SPs who are out of sync 3. Devs / analysts who need historical data

 Note: This is an easy data use case because there are clear users and SPs necessarily have the right data on hand. If we can get this model working it is easy to imagine extending it to many different classes of data (i.e. other blockchain, public data set).  A decent replacement for FIL+ is trusting operators with an endowment of funds to bless data and pay out for it using this model.

 a. Software for incremental snapshot. New daemon holds everything in memory with ref counting listens to lotus head change to increment in memory snapshot. Need lotus api 60-100G machines to run, more in nv19 style crises. When we need to make a snapshot we flush from mem to disk

 Slightly more detailed breakdown

 Fil-fil process snapshotting 
 This is a separate process (`fil-fil`...) that listens to a lotus node api. 
 The lotus head change api is the event that drives changes to this processes internal state
 This process maintains an in memory mapping of all cid -> data pairs that belong in the snapshot and the latest head
 When there is a new head it adds new state and recycles state that is now too far in the past to be in the snapshot
 Track all block headers coming from lotus-api.  `saaf.Link` new headers.  `saaf.Unlink` old headers.
 I wrote a very simple go library for doing this efficiently: https://github.com/ZenGround0/saaf, I wrote it with the intention of using it in this project
 When `fil-fil` daemon is told by the user to generate a snapshot i.e. `fil-fil take-snapshot` it does a car walk over the map, but in memory.  This should make generating the snapshot take order of minutes (ideally seconds) instead of hours when going to disk.  I haven’t done much performance analysis so maybe it is still fairly slow and we’ll need something more sophisticated.
 The point of this design is that we can very quickly generate snapshots when we need them.  This allows us to satisfy the user requirement of real time snapshot storage
 Note that this software would generally be useful even if we don’t create an overlay data incentivization network.  It should be 10x (at least) faster than current go-to-disk snapshot solutions in lotus 
 	

 b. Deal making thing, probably in the same daemon. Integrates with Boost, knows about block time and when its slot is
 Providers running fil-fil are elected at a regular interval to add a snapshot at a certain epoch.  They know well in advance when this is 
 `fil-fil store <snapshot-epoch-number>` sends a client-contract deal to boost which then communicates to lotus-miner to send the data for sealing.  Since snapshots are over 64 GiB it will be two pieces.  `fil-fil store` could directly be taking the snapshot saving the file and then talking to boost or perhaps it is separate from `fil-fil take-snapshot` and takes in the snapshot file.
 The deal’s client is the fil-fil smart contract
 We’ll want this to be a snapdeal to commit snapshots quickly, im not sure how to communicate that to the lotus-miner process there is some work involved there

 	

 c. Deal client smart contract. 
 1. The contract handles registering of SP addresses as Fil Fil providers.  
 This is a bit of an open problem.  You could register by posting a bit of collateral, it could be free for anyone to join, there could be a registration limit so that only a small fixed number is allowed at any one time
 Registration should be revoked if you miss a certain amount of snapshots or if your snapshots are terminated early.
 Trivial to sybil round robin if there is no collateral.  
 Only reason to worry about this is that we want a diverse set of participants to store snapshots not just one SP
 There are limits to collateral, if one SP really wants all the slots they can just pay a lot. 
 It could be nice to limit participation so that there is some floor on the deal revenue that participants are getting – new joiners can’t come in and make running snapshotting 

 2. Registered providers are elected in a round-robin manner.  The contract can tell when it is the provider’s turn to make a snapshot deal and this is when the provider calls `fil-fil store` after the contact operator posts the commP for their snapshot slot.  `fil-fil next window` will tell the first / last epochs that a provider can claim fil-fil deals for snapshots.
 Round robin scheduling is tricky and is related to registration
 New registrations should not make new snapshot slots unpredictable.  We could achieve this by doing rounds of everyone being elected, and new registrants are slotted in for the next round. 
 If you miss some rounds (lets say 3 arbitrarily) then youre address is no longer registered

 3. The main functionality is deal-client-contract.  The contract pays out if deal passes these validation conditions:
 The provider is registered
 It is currently the provider’s slot
 The commP matches the staged commp created by the operator
 		a. Validating these conditions must be done in a typical efficient smart contract pattern.  There should be a very small amount of data to be read for a provider to know if they are in their window.  A solidity map from registered provider address to next window start and end epochs that is updated on client contract notification should do the trick.


 	4.Nice to have – deregistering SPs that either 1) terminate their deals early 2) miss their round robin window.
 		


 d. A trusted party runs fil-fil as an operator to publish commP.  This instance of fil-fil has a special key that is trusted to bless data valid for activating as a deal

 The simplest idea is for the contract to have a permissioned owner.  Some multisig account with trusted keys that are highly protected
 We could also do something crazy and have DAO like voting.  That is probably more complicated than v1 should be.
 The operator runs `fil-fil incentivize <epoch-of-snapshot>` themselves to get the snapshot hash and publishes it ahead of fil-fil participant to unlock rewards for this hash
 We don’t need to worry too much about reorgs since reorged snapshots are still very useful.  We could even just publish immediately and have `fil-fil` process keep track of several latest heads before pruning reorgs so that all participants will have the right head	
 Reorg stuff needs more thought.  General tradeoff between complexity and speed.  

 e. get contract FIL+ notarized
 If FIL+ is deprecated in the near future we would need to apply for a grant for funds.  We could also do something more crazy like issue a DAO token
 Getting this contract notarized with lots of datacap should not be that hard assuming FIL+ is still active


 f.Plug fil-fil providers into the Saturn network to provide snapshot data.  
 If fil-fil network participants are also saturn nodes then they can serve snapshot data immediately after generation (ideally < 1 minute) well before snap deal sector committing is finished.
 Fil-fil participants could run saturn nodes on snapshot data separate from the filecoin storage aspect.  There will be no way to guarantee this however.  Not until retrieval guarantee primitives are invented.
 The fil-fil contract can fairly easily be written in a way to provide a public index of who is storing the latest snapshots / snapshots at which time and so public information about retrieval can be accessed there.  This is maybe useful. 
 Much more research needs to be done here since I do not know much about saturn network.  It seems like a good direction however
 We can always use basic filecoin retrieval among fil-fil providers though we’ll want to figure out payments, maybe there are good simple hacks we can figure out to get most of the way there without using a full content delivery network
	Fil_Fil Product:
	a small incentivized data network overlay on top of filecoin specifically for filecoin snapshots
	User requirement: snapshots are incentivized to be stored for a long time, ~2-5 years with replication
	User requirement: snapshots are committed in real time < 1 hour after they are generated
	User nice-to-have: snapshots are available for content delivery on the saturn network

	Big idea: providers participate in the network for an incentive. Users of the network get guarantee that snapshot data is stored and (ideally) a reliable way to get the snapshot data out if they pay. Basically it is like filecoin but for a tiny subset of specific data.

	Users: 1. People installing lotus who need to sync (ideally FilecoinFoundation would subsiize saturn payments for people downloading from lotus-chain management website within reason 2. SPs who are out of sync 3. Devs / analysts who need historical data

	Note: This is an easy data use case because there are clear users and SPs necessarily have the right data on hand. If we can get this model working it is easy to imagine extending it to many different classes of data (i.e. other blockchain, public data set). A decent replacement for FIL+ is trusting operators with an endowment of funds to bless data and pay out for it using this model.

	a. Software for incremental snapshot. New daemon holds everything in memory with ref counting listens to lotus head change to increment in memory snapshot. Need lotus api 60-100G machines to run, more in nv19 style crises. When we need to make a snapshot we flush from mem to disk

	Slightly more detailed breakdown

	Fil-fil process snapshotting
	This is a separate process (`fil-fil`...) that listens to a lotus node api.
	The lotus head change api is the event that drives changes to this processes internal state
	This process maintains an in memory mapping of all cid -> data pairs that belong in the snapshot and the latest head
	When there is a new head it adds new state and recycles state that is now too far in the past to be in the snapshot
	Track all block headers coming from lotus-api. `saaf.Link` new headers. `saaf.Unlink` old headers.
	I wrote a very simple go library for doing this efficiently: https://github.com/ZenGround0/saaf, I wrote it with the intention of using it in this project
	When `fil-fil` daemon is told by the user to generate a snapshot i.e. `fil-fil take-snapshot` it does a car walk over the map, but in memory. This should make generating the snapshot take order of minutes (ideally seconds) instead of hours when going to disk. I haven’t done much performance analysis so maybe it is still fairly slow and we’ll need something more sophisticated.
	The point of this design is that we can very quickly generate snapshots when we need them. This allows us to satisfy the user requirement of real time snapshot storage
	Note that this software would generally be useful even if we don’t create an overlay data incentivization network. It should be 10x (at least) faster than current go-to-disk snapshot solutions in lotus


	b. Deal making thing, probably in the same daemon. Integrates with Boost, knows about block time and when its slot is
	Providers running fil-fil are elected at a regular interval to add a snapshot at a certain epoch. They know well in advance when this is
	`fil-fil store <snapshot-epoch-number>` sends a client-contract deal to boost which then communicates to lotus-miner to send the data for sealing. Since snapshots are over 64 GiB it will be two pieces. `fil-fil store` could directly be taking the snapshot saving the file and then talking to boost or perhaps it is separate from `fil-fil take-snapshot` and takes in the snapshot file.
	The deal’s client is the fil-fil smart contract
	We’ll want this to be a snapdeal to commit snapshots quickly, im not sure how to communicate that to the lotus-miner process there is some work involved there



	c. Deal client smart contract.
	1. The contract handles registering of SP addresses as Fil Fil providers.
	This is a bit of an open problem. You could register by posting a bit of collateral, it could be free for anyone to join, there could be a registration limit so that only a small fixed number is allowed at any one time
	Registration should be revoked if you miss a certain amount of snapshots or if your snapshots are terminated early.
	Trivial to sybil round robin if there is no collateral.
	Only reason to worry about this is that we want a diverse set of participants to store snapshots not just one SP
	There are limits to collateral, if one SP really wants all the slots they can just pay a lot.
	It could be nice to limit participation so that there is some floor on the deal revenue that participants are getting – new joiners can’t come in and make running snapshotting

	2. Registered providers are elected in a round-robin manner. The contract can tell when it is the provider’s turn to make a snapshot deal and this is when the provider calls `fil-fil store` after the contact operator posts the commP for their snapshot slot. `fil-fil next window` will tell the first / last epochs that a provider can claim fil-fil deals for snapshots.
	Round robin scheduling is tricky and is related to registration
	New registrations should not make new snapshot slots unpredictable. We could achieve this by doing rounds of everyone being elected, and new registrants are slotted in for the next round.
	If you miss some rounds (lets say 3 arbitrarily) then youre address is no longer registered

	3. The main functionality is deal-client-contract. The contract pays out if deal passes these validation conditions:
	The provider is registered
	It is currently the provider’s slot
	The commP matches the staged commp created by the operator
	a. Validating these conditions must be done in a typical efficient smart contract pattern. There should be a very small amount of data to be read for a provider to know if they are in their window. A solidity map from registered provider address to next window start and end epochs that is updated on client contract notification should do the trick.


	4.Nice to have – deregistering SPs that either 1) terminate their deals early 2) miss their round robin window.



	d. A trusted party runs fil-fil as an operator to publish commP. This instance of fil-fil has a special key that is trusted to bless data valid for activating as a deal

	The simplest idea is for the contract to have a permissioned owner. Some multisig account with trusted keys that are highly protected
	We could also do something crazy and have DAO like voting. That is probably more complicated than v1 should be.
	The operator runs `fil-fil incentivize <epoch-of-snapshot>` themselves to get the snapshot hash and publishes it ahead of fil-fil participant to unlock rewards for this hash
	We don’t need to worry too much about reorgs since reorged snapshots are still very useful. We could even just publish immediately and have `fil-fil` process keep track of several latest heads before pruning reorgs so that all participants will have the right head
	Reorg stuff needs more thought. General tradeoff between complexity and speed.

	e. get contract FIL+ notarized
	If FIL+ is deprecated in the near future we would need to apply for a grant for funds. We could also do something more crazy like issue a DAO token
	Getting this contract notarized with lots of datacap should not be that hard assuming FIL+ is still active


	f.Plug fil-fil providers into the Saturn network to provide snapshot data.
	If fil-fil network participants are also saturn nodes then they can serve snapshot data immediately after generation (ideally < 1 minute) well before snap deal sector committing is finished.
	Fil-fil participants could run saturn nodes on snapshot data separate from the filecoin storage aspect. There will be no way to guarantee this however. Not until retrieval guarantee primitives are invented.
	The fil-fil contract can fairly easily be written in a way to provide a public index of who is storing the latest snapshots / snapshots at which time and so public information about retrieval can be accessed there. This is maybe useful.
	Much more research needs to be done here since I do not know much about saturn network. It seems like a good direction however
	We can always use basic filecoin retrieval among fil-fil providers though we’ll want to figure out payments, maybe there are good simple hacks we can figure out to get most of the way there without using a full content delivery network