Inspecting proposals with PyTezos

dyor_voting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Current voting phase\n",
    "\n",
    "Say, you don't read any news and all you have is public node access and documentation.  \n",
    "How to determine what is the current voting phase?  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytezos.rpc import mainnet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'proposal'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mainnet.head.votes.current_period_kind()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great, we are at the first stage, now we want to know when did it start and when it will ends.  \n",
    "We can get this information from the block metadata:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'level': 349439,\n",
       " 'level_position': 349438,\n",
       " 'cycle': 85,\n",
       " 'cycle_position': 1278,\n",
       " 'voting_period': 10,\n",
       " 'voting_period_position': 21758,\n",
       " 'expected_commitment': False}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "level_info = mainnet.head.metadata.get('level')\n",
    "level_info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get precise boundaries in levels and rough estimation in days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pendulum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "327681"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start_level = level_info['level'] - level_info['voting_period_position']\n",
    "start_level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "360448"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "end_level = start_level + 8 * 4096 - 1  # eight cycles of 4096 blocks\n",
    "end_level"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "15.7 days passed\n"
     ]
    }
   ],
   "source": [
    "start_dt = pendulum.parse(mainnet.blocks[start_level].header.get('timestamp'))\n",
    "time_past = (pendulum.now() - start_dt)\n",
    "print(round(time_past.total_days(), 1), 'days passed')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.9 days left\n"
     ]
    }
   ],
   "source": [
    "time_left = (end_level - level_info['level']) / (level_info['level'] - start_level) * time_past\n",
    "print(round(time_left.total_days(), 1), 'days left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Current proposals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z', 5176],\n",
       " ['Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd', 13255]]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proposals = mainnet.head.votes.proposals()\n",
    "proposals"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine one of proposals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proposal_id = proposals[1][0]\n",
    "proposal_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Injection operation\n",
    "\n",
    "It's interesting to find the author of the proposal.  \n",
    "In order to do that we have to search for the first `proposals` operation for this particular proposal_id."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytezos.rpc.search import SearchChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc = SearchChain.from_chain(mainnet.main)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to the statefullness of the Tezos blockchain we can perform a binary search inside the voting period.  \n",
    "The algorithm searches for the first level where number of votes changed from 0 to non-zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-03-12 19:40:38.314 | DEBUG    | pytezos.rpc.search:bisect:29 - 714 at level 338560\n",
      "2019-03-12 19:40:38.905 | DEBUG    | pytezos.rpc.search:bisect:29 - 23 at level 333120\n",
      "2019-03-12 19:40:39.499 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 330400\n",
      "2019-03-12 19:40:40.051 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 331760\n",
      "2019-03-12 19:40:40.602 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 332440\n",
      "2019-03-12 19:40:41.198 | DEBUG    | pytezos.rpc.search:bisect:29 - 11 at level 332780\n",
      "2019-03-12 19:40:41.748 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 332610\n",
      "2019-03-12 19:40:42.306 | DEBUG    | pytezos.rpc.search:bisect:29 - 11 at level 332695\n",
      "2019-03-12 19:40:43.092 | DEBUG    | pytezos.rpc.search:bisect:29 - 11 at level 332652\n",
      "2019-03-12 19:40:43.898 | DEBUG    | pytezos.rpc.search:bisect:29 - 11 at level 332631\n",
      "2019-03-12 19:40:44.770 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 332620\n",
      "2019-03-12 19:40:45.598 | DEBUG    | pytezos.rpc.search:bisect:29 - 11 at level 332625\n",
      "2019-03-12 19:40:46.410 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 332622\n",
      "2019-03-12 19:40:47.229 | DEBUG    | pytezos.rpc.search:bisect:29 - 0 at level 332623\n",
      "2019-03-12 19:40:47.999 | DEBUG    | pytezos.rpc.search:bisect:29 - 11 at level 332624\n"
     ]
    }
   ],
   "source": [
    "operation = sc.find_proposal_inject_operation(proposal_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'protocol': 'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP',\n",
       " 'chain_id': 'NetXdQprcVkpaWU',\n",
       " 'hash': 'onydFJLWdGhfKNBfbnSLmqDu93j9NRimkbQm9WqLWYG8eyZUyTF',\n",
       " 'branch': 'BL53WJx6xPn6rnTnWZmpNaWGAQqU8HTwRTDqVDmthYsxUTBewo9',\n",
       " 'contents': [{'kind': 'proposals',\n",
       "   'source': 'tz1fNdh4YftsUasbB1BWBpqDmr4sFZaPNZVL',\n",
       "   'period': 10,\n",
       "   'proposals': ['Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd',\n",
       "    'Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z'],\n",
       "   'metadata': {}}],\n",
       " 'signature': 'sigvUqvh7rBS8yAoE5RMieQaD5hvg9NsLeJ4kTnXdK1tXXyrHL8mX7E3KCm9q9YgYbJn3edhcUiZjdU3xNhVPEUPkSGVNbi9'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "operation()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can perform some checks on this operation, such as signature validation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "operation.verify_signature()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only thing we can learn about the submitter's identity is his public key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'edpktyybTTrLm2rk6bn7xtgY9t2Kgt9GnqZxEcSrunm4vKWTF9ES9o'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mainnet.get_public_key(operation.source())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Proposed on the 4th day of the voting period'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timestamp = mainnet.blocks[operation.get('branch')].header.get('timestamp')\n",
    "inject_dt = pendulum.parse(timestamp)\n",
    "f'Proposed on the {(inject_dt - start_dt).days + 1}th day of the voting period'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Source code\n",
    "\n",
    "Unfortunately it's practically impossible to get proposal source from the blockchain.  \n",
    "We could get lucky and find the submitter's node (in case it's public) which has to know this proto (according to the docs).  \n",
    "The other option is trying to find a node in zeronet which has participated in the voting rehearsal.  \n",
    "\n",
    "We will be back to this issue later, now let's download proposal sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "athens_a_url = 'https://blog.nomadic-labs.com/files/Athens_proposal_A.tar'\n",
    "athens_b_url = 'https://blog.nomadic-labs.com/files/Athens_proposal_B.tar'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytezos.rpc.protocol import Protocol"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loading sources and convert them to the internal format (as in blockchain).  \n",
    "This function can also import a local tar file or a directory with extracted files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1218560it [03:37, 5613.44it/s]\n"
     ]
    }
   ],
   "source": [
    "athens_a = Protocol.from_uri(athens_a_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "1218560it [03:04, 6594.80it/s]\n"
     ]
    }
   ],
   "source": [
    "athens_b = Protocol.from_uri(athens_b_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that the sources we have downloaded are original.  \n",
    "In order to do that we have to obtain binary representation of the sources according to the http://tezos.gitlab.io/mainnet/api/rpc.html#get-protocols-protocol-hash (Binary output tab)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "athens_a_id = athens_a.calculate_hash()\n",
    "athens_a_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Psd1ynUBhMZAeajwcZJAeq5NrxorM6UCU4GJqxZ7Bx2e9vUWB6z'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "athens_b_id = athens_b.calculate_hash()\n",
    "athens_b_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert athens_a_id == proposals[1][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert athens_b_id == proposals[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Protocol update diff\n",
    "\n",
    "First of all we need to get sources of the current protocol:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "protocols/PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_proto = mainnet.protocols[operation.get('protocol')]\n",
    "current_proto"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "current_hash = current_proto.calculate_hash()\n",
    "current_hash"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert current_hash == operation.get('protocol')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can generate patch file in the standard diff format.  \n",
    "We can optional specify number of lines before and after the change: this is useful for review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "patch = current_proto.diff(athens_a, context_size=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate github-like side-by-side patch views, powered by diff2html.js"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "patch.export_html(output_path='athens_a_vs_003.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Compare \"Athens A\" vs current](https://baking-bad.github.io/tezos-dyor/athens_a_vs_003_PsddFKi3.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "athens_diff = athens_a.diff(athens_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "athens_diff.export_html(output_path='athens_a_vs_athens_b.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Compare \"Athens A\" vs \"Athens B\"](https://baking-bad.github.io/tezos-dyor/athens_a_vs_athens_b.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bonus: get all voting operations for a proposal\n",
    "\n",
    "As an alternative to TzScan and other indexed-blockchain solutions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2019-03-12 19:47:43.747 | DEBUG    | pytezos.rpc.search:find_state_change_intervals:14 - 13255 at level 349385\n",
      "2019-03-12 19:47:44.360 | DEBUG    | pytezos.rpc.search:find_state_change_intervals:14 - 13255 at level 349325\n",
      "2019-03-12 19:47:45.133 | DEBUG    | pytezos.rpc.search:find_state_change_intervals:14 - 13183 at level 349265\n",
      "2019-03-12 19:47:45.711 | DEBUG    | pytezos.rpc.search:bisect:29 - 13183 at level 349295\n",
      "2019-03-12 19:47:46.308 | DEBUG    | pytezos.rpc.search:bisect:29 - 13255 at level 349310\n",
      "2019-03-12 19:47:46.911 | DEBUG    | pytezos.rpc.search:bisect:29 - 13255 at level 349302\n",
      "2019-03-12 19:47:47.482 | DEBUG    | pytezos.rpc.search:bisect:29 - 13183 at level 349298\n",
      "2019-03-12 19:47:48.127 | DEBUG    | pytezos.rpc.search:bisect:29 - 13183 at level 349300\n",
      "2019-03-12 19:47:48.715 | DEBUG    | pytezos.rpc.search:bisect:29 - 13255 at level 349301\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'protocol': 'PsddFKi32cMJ2qPjf43Qv5GDWLDPZb3T3bF6fLKiF5HtvHNU7aP', 'chain_id': 'NetXdQprcVkpaWU', 'hash': 'ooRoJ2yiEqchbV4xWEGk3wjwJtm5A5GcUnLFmYNLRzUgTT52ffU', 'branch': 'BLoUbPrEGth1WPJ4uFwEVTp5eWKqUU8Jj9EHqrsndmN9xE1QxiE', 'contents': [{'kind': 'proposals', 'source': 'tz1dwu9aYb7CRNq4Y2zAjipdjFuSVKhHS8vA', 'period': 10, 'proposals': ['Pt24m4xiPbLDhVgVfABUjirbmda3yohdN82Sp9FeuAXJ4eV9otd'], 'metadata': {}}], 'signature': 'sigboCbwcDzNJQXEPyCYfoL6fd5oBeTA2ogfhydhVEN9agdZg6PzktCZ4Fvj4WA5X8RUwz3t9jxS5gGftXJcF8R5tvNrbhNd'}\n"
     ]
    }
   ],
   "source": [
    "for operation in sc.find_proposal_votes_operations(proposal_id):\n",
    "    print(operation())\n",
    "    break  # this can take a while"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Search algorithm works as follows:\n",
    "1. Split block interval into equal chunks\n",
    "2. Determine which of the intervals contain state changes\n",
    "3. For each interval run binary search\n",
    "4. If there are several changes inside single interval, run binary search again\n",
    "\n",
    "It's obvious that the search space can be easily splitted and processed independently, i.e parallelized. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Improvement: proposal sources on-chain\n",
    "\n",
    "As we pointed earlier there is no convenient way to get proposal source from the blockchain. This can be implemented via smart-contract. But it's more reasonable to store compressed code diff rather than full source."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctxless_patch = current_proto.diff(athens_a, context_size=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "ctxless_patch.export_tar('diff.tar.bz2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12839"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.getsize('diff.tar.bz2')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying protocol diff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "proto = current_proto.apply(ctxless_patch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert proposal_id == proto.calculate_hash()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python 3.7",
   "language": "python",
   "name": "python36"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}