Skip to content

Instantly share code, notes, and snippets.

@m-kus

m-kus/fa_token_balance_updates.ipynb

Last active May 31, 2020 10:52
Show Gist options
  • Save m-kus/a9081b75d91d6cba9ed1cfe8b1b59e40 to your computer and use it in GitHub Desktop.
Save m-kus/a9081b75d91d6cba9ed1cfe8b1b59e40 to your computer and use it in GitHub Desktop.
Download ZIP
Deriving FA token balance_updates from big_map_diff
Display the source blob
Display the rendered blob
Raw
fa_token_balance_updates.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "tzbtc_big_map_diff_parser.ipynb",
"provenance": [],
"collapsed_sections": []
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "bWxVdhUboRVx",
"colab_type": "text"
},
"source": [
"# Deriving FA token `balance_updates` from `big_map_diff`\n",
"\n",
"### Why?\n",
"There are two separate issues while indexing token operations:\n",
"1. We need some basic metadata to display balances, at least `symbol` and number of `decimals`;\n",
"2. In `FA1.2` and upcoming `FA2` only `transfer` method is standardized, but there are other methods that alter token balances such as `mint` and `burn`; thus we cannot do accounting based on `transfer` call parameters.\n",
"\n",
"### What we need\n",
"A lightweight, and preferably stateless solution that could derive token balance updates from just RPC output (`/chains/main/blocks/{block_id}`).\n",
"This would allow to add **generic** token support to the standard indexer workflow at minimal cost. Ideally, it should work like the `balance_updates` receipts.\n",
"\n",
"### Big Map Diff\n",
"It can be argued that the vast majority of contracts implementing FA-family standards are/will store its balance ledgers in lazy structures aka `Big_map`. \n",
"Thus, the `big_map_diff` data (list of changed `Big_map` keys with new values) that is being attached to every operation is actually what we need. The only problem is that different contracts have different storage types and we need to know exactly how to extract the data we need.\n",
"\n",
"### Custom handlers\n",
"As mentioned above, in addition to FA-standardized methods, there may be other methods that modify user balances and it is impossible to standardize them all, plus we would have to sacrifice flexibility. \n",
"This means that indexer developers have to implement custom handlers for each new FA token (this is how it works at the moment). It's not good for many reasons, and there should be another solution.\n",
"\n",
"### Michelson scripts/plugins\n",
"The idea of using Michelson as a generic script language has been voiced several times [1], [2] and seems like a way out. But the following question arises: how to execute these scripts. Of course you can use the standard RPC `run_code` endpoint but it's costly and suboptimal. \n",
"Luckily, there's an ongoing work on encapsulating the Michelson script interpreter and hopefully we will be able to link it as a standalone library in the future. Moreover, there are several other Michelson implementations (in Haskell by Serokell, in Python by Baking Bad, probably more), and since we need only a relatively small subset of instructions (no blockchain bindings which are most costly to implement) it's a rather doable task to write an own, or to make a collaborative effort. \n",
"**NOTE:** You can actually write a parser script in any high-level language you want (LIGO, SmartPy, Lorentz, SCaml, etc.) since it's a valid Tezos contract and can be compiled down.\n",
"\n",
"### Possible Shell integration\n",
"Another alternative to writing own interpreter is to integrate this plugin system into the Tezos Shell. It would require to register custom `big_map_diff` parsers by the node operator, e.g. like that: \n",
"```\n",
"tezos-node register big_map_diff handler \"tzbtc_parser.tz\" for big_map 31\n",
"```\n",
"From the POV of a developer it could look like an RPC response extension, for each supported `Big_map` he would receive an extra operation receipt:\n",
"```\n",
"...\n",
"\"operation_result\": {\n",
" \"balance_updates\": [\n",
" {\n",
" \"kind\": \"token\",\n",
" \"address\": \"tz1d75oB6T4zUMexzkr5WscGktZ1Nss1JrT7\",\n",
" \"balance\": 100500,\n",
" \"symbol\": \"TZBTC\",\n",
" \"decimals\": 8\n",
" }, \n",
" ...\n",
" ]\n",
"}\n",
"...\n",
"```\n",
"\n",
"### Other questions\n",
"#### Who will write parser scripts\n",
"Token developers. Many scripts would likely be reusable (especially in case of contract factories).\n",
"\n",
"#### Where to store parser scripts\n",
"It seems logical to keep them on-chain since they are actually valid contracts. FA token contracts can keep the parser address (pointer) in the storage.\n",
"\n",
"#### Could there be multiple parsers\n",
"The approach can be generalized to extract any other data and not just from `big_map_diff`, but also from contract storage, operation parameters. You can basically move all view methods that are not used by other contracts off-chain, make them external (as suggested in [1]).\n",
"\n",
"### Proof-of-concept\n",
"A parser script for the TZBTC contract.\n",
"A step-by-step tutorial is available at https://nbviewer.jupyter.org/github/baking-bad/michelson-kernel/blob/binder/tzbtc_big_map_diff_parser.ipynb#result \n",
"Or try it out in [Binder](https://mybinder.org/v2/gh/baking-bad/michelson-kernel/binder?filepath=tzbtc_big_map_diff_parser.ipynb) \n",
"**NOTE:** This is probably the hardest case one could imagine, most scripts would be much much simpler.\n",
"\n",
"### Inspired by\n",
"[1] https://forum.tezosagora.org/t/external-views/1768 by Gabriel Alfour \n",
"[2] https://smondet.gitlab.io/-/fa2-smartpy/-/jobs/568428144/artifacts/testweb/tutorial.html#get-balance-off-chain by Sebastien Mondet\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "SehcnxqTB33T",
"colab_type": "text"
},
"source": [
"# Demo"
]
},
{
"cell_type": "code",
"metadata": {
"id": "P-vPQ3ipoMBr",
"colab_type": "code",
"colab": {}
},
"source": [
"!apt install libsodium-dev libsecp256k1-dev libgmp-dev"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "SWaHpRRNnR3l",
"colab_type": "code",
"colab": {}
},
"source": [
"!pip install pytezos"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "610Fm4QnnTJy",
"colab_type": "code",
"colab": {}
},
"source": [
"from pytezos import pytezos, Contract\n",
"from pytezos.operation.result import OperationResult\n",
"from pytezos.repl.interpreter import Interpreter\n",
"from pytezos.michelson.converter import micheline_to_michelson, michelson_to_micheline\n",
"from pytezos.michelson.interface import ContractCallResult\n",
"from pprint import pprint"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "p3SRunk8nVHx",
"colab_type": "code",
"colab": {}
},
"source": [
"pytezos = pytezos.using('mainnet')"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "m1FbXgRtnWtE",
"colab_type": "code",
"colab": {}
},
"source": [
"transfer_opg = pytezos.shell.blocks[970528].operations['opPrFk4dUdefgXs14fh4xyBpuSzrMWSv8HWWgEei3oc4qmoW8wX']()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "2W2Q6u4InZdY",
"colab_type": "code",
"colab": {}
},
"source": [
"transfer_res = OperationResult.from_operation_group(transfer_opg)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "vz9hkZionc7S",
"colab_type": "code",
"outputId": "18d65e11-4f66-4e00-abf2-8f1f76d09a31",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 283
}
},
"source": [
"transfer_res[0].big_map_diff"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"[{'action': 'update',\n",
" 'big_map': '31',\n",
" 'key': {'bytes': '05010000000b746f74616c537570706c79'},\n",
" 'key_hash': 'exprunzteC5uyXRHbKnqJd3hUMGTWE9Gv5EtovDZHnuqu6SaGViV3N',\n",
" 'value': {'bytes': '050098e1e8d78a02'}},\n",
" {'action': 'update',\n",
" 'big_map': '31',\n",
" 'key': {'bytes': '05070701000000066c65646765720a00000016000016e64994c2ddbd293695b63e4cade029d3c8b5e3'},\n",
" 'key_hash': 'expruiaeokjY8rPY52YXKZ6zK7oBN9Cx52psQyHtup13vMUmM7e4X2',\n",
" 'value': {'bytes': '05070700ac9a010200000000'}},\n",
" {'action': 'update',\n",
" 'big_map': '31',\n",
" 'key': {'bytes': '05070701000000066c65646765720a000000160000bf97f5f1dbfd6ada0cf986d0a812f1bf0a572abc'},\n",
" 'key_hash': 'expruzQsCpesXXLyKhYwQdFLy48FjXCd2T7xQ5xioevvC1pUhv3rAN'}]"
]
},
"metadata": {
"tags": []
},
"execution_count": 22
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "FHPYGzfinf7s",
"colab_type": "code",
"colab": {}
},
"source": [
"diffs = list(filter(lambda x: x['action'] == 'update' and x['big_map'] == '31', \n",
" transfer_res[0].big_map_diff))"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "-GBoRtOAnjLe",
"colab_type": "code",
"colab": {}
},
"source": [
"parser_script = Contract.from_michelson(\"\"\"\n",
"parameter (pair %decodeBigMapDiff (bytes %key) (option (bytes %value)));\n",
"storage (map (address %holder) \n",
" (pair (nat %balance) \n",
" (pair (nat %decimals) \n",
" (string %symbol))));\n",
"code {\n",
" DUP ; CAR ; DIP { CDR } ;\n",
" DUP ; CAR ;\n",
" UNPACK (pair string address) ;\n",
" IF_SOME { \n",
" DUP ; CAR ;\n",
" PUSH string \"ledger\" ;\n",
" IFCMPEQ { \n",
" CDR @holder ; \n",
" SWAP ; CDR ;\n",
" IF_SOME { \n",
" UNPACK (pair nat (map address nat)) ;\n",
" ASSERT_SOME ;\n",
" CAR @balance ;\n",
" } { \n",
" PUSH @balance nat 0;\n",
" } ;\n",
" PUSH @symbol string \"TZBTC\" ;\n",
" PUSH @decimals nat 8 ;\n",
" PAIR ; SWAP ; PAIR ; SOME ;\n",
" SWAP ; UPDATE \n",
" } {\n",
" DROP 2\n",
" } ;\n",
" } { \n",
" DROP\n",
" } ;\n",
" NIL operation ; PAIR ;\n",
"}\n",
"\"\"\")"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "v_PjHXgVnoO8",
"colab_type": "code",
"colab": {}
},
"source": [
"i = Interpreter(debug=False)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "a3CDfygCnqTo",
"colab_type": "code",
"outputId": "1f26fe96-dd43-4a8e-e4d3-d5fecbe5907f",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 94
}
},
"source": [
"i.execute(parser_script.text)"
],
"execution_count": 0,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/plain": [
"{'result': None,\n",
" 'stdout': 'parameter (pair %decodeBigMapDiff (bytes %key) (option (bytes %value)));\\nstorage (map (address %holder) (pair (nat %balance) (pair (nat %decimals) (string %symbol))));\\ncode { DUP ; CAR ; DIP { CDR } ; DUP ; CAR ; UNPACK (pair string address) ; { IF_NONE { DROP } { DUP ; CAR ; PUSH string \"ledger\" ; { { COMPARE ; EQ } ; IF { CDR @holder ; SWAP ; CDR ; { IF_NONE { PUSH @balance nat 0 } { UNPACK (pair nat (map address nat)) ; IF_NONE { { UNIT ; FAILWITH } } {} ; CAR @balance } } ; PUSH @symbol string \"TZBTC\" ; PUSH @decimals nat 8 ; PAIR ; SWAP ; PAIR ; SOME ; SWAP ; UPDATE } { DROP 2 } } } } ; NIL operation ; PAIR };',\n",
" 'success': True}"
]
},
"metadata": {
"tags": []
},
"execution_count": 26
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "cP2yaincnsZH",
"colab_type": "code",
"colab": {}
},
"source": [
"def diff_to_token_update(diff):\n",
" key = micheline_to_michelson(diff['key'], inline=True, wrap=True)\n",
" value = micheline_to_michelson(diff['value'], inline=True, wrap=True) if diff.get('value') else None\n",
" val = f'(Some {value})' if value else 'None'\n",
" params = f'(Pair {key} {val})'\n",
" res = i.execute(f'RUN {params} {{}}')\n",
" call_res = ContractCallResult.from_repl_result(res, michelson_to_micheline(params), parser_script)\n",
" return call_res.storage"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "iW2P3Li-n7HY",
"colab_type": "code",
"outputId": "06f71501-5d8f-4791-b9cc-7b2e63a4ccf5",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 150
}
},
"source": [
"pprint(list(map(diff_to_token_update, diffs)))"
],
"execution_count": 0,
"outputs": [
{
"output_type": "stream",
"text": [
"[{},\n",
" {'tz1Mj7RzPmMAqDUNFBn5t5VbXmWW4cSUAdtT': {'balance': 9900,\n",
" 'decimals': 8,\n",
" 'symbol': 'TZBTC'}},\n",
" {'tz1d75oB6T4zUMexzkr5WscGktZ1Nss1JrT7': {'balance': 0,\n",
" 'decimals': 8,\n",
" 'symbol': 'TZBTC'}}]\n"
],
"name": "stdout"
}
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment