Skip to content

Instantly share code, notes, and snippets.

@jeethu

jeethu/live_signals_sharpe_sortino.ipynb

Last active July 8, 2021 17:22
Show Gist options
  • Save jeethu/5da476a91060a3fea567cf489396295a to your computer and use it in GitHub Desktop.
Save jeethu/5da476a91060a3fea567cf489396295a to your computer and use it in GitHub Desktop.
Download ZIP
Live annualized Sharpe and Sortino metrics for Numerai Signals
Display the source blob
Display the rendered blob
Raw
live_signals_sharpe_sortino.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a href=\"https://colab.research.google.com/gist/jeethu/5da476a91060a3fea567cf489396295a\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "iN0nNtRKWMyS"
},
"source": [
"# Live annualized Sharpe and Sortino metrics for Numerai Signals\n",
"Copyright Degerhan Usluel, license: [MIT](https://opensource.org/licenses/MIT)\n",
"\n",
"Adapted for Numerai Signals (from Degerhan's Numerai classic [notebook](https://gist.github.com/degerhan/172be51b80c3d9b218d3d46c5fad6a69)) by [jrb](https://numer.ai/jrb)."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "CxCtAsPdWMyV"
},
"source": [
"## The value of daily scores\n",
"\n",
"Your numerai models earn or burn when a round is resolved. The 20 daily scores you receive in between are for kicks and giggles, and don't mean anything.\n",
"\n",
"Or do they? A model with a slow and steady climb might be more deserving of your hard earned NMRs, even when other models resolve with higher corr and mmc scores.\n",
"\n",
"The historical daily scores are also a high resolution indicator of your model's volatility under different staking regimes, and may influence your choice of stake type.\n",
"\n",
"This notebook calculates the Sharpe and Sortino ratios for a list of models, for the four different staking regimes, based on their daily gyrations."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "yt1iJX87WMyX"
},
"source": [
"## Scaling daily return series to annual Sharpe ratio\n",
"\n",
"Published hedge fund Sharpe ratios are annualized numbers. Same for mutual funds, see [Morningstar Methodology Paper](https://gladmainnew.morningstar.com/directhelp/Methodology_StDev_Sharpe.pdf).\n",
"\n",
"The financial industry practice is to start with monthly return series. Often arithmetic average is used for returns (12 x monthly mean return) rather than geometric, and monthly series standard deviation is time-scaled to annual by multiplying with $\\sqrt{12}$.\n",
"\n",
"Andrew Lo warns us about the perils of serial correlations and multiplying by $\\sqrt{12}$ in [Statistics of Sharpe Ratios](https://alo.mit.edu/wp-content/uploads/2017/06/The-Statistics-of-Sharpe-Ratios.pdf) ). Yet, warts and all, this is the metric every fund publishes, so we shamelessly use $\\sqrt{252}$ to time-scale from daily to annualized volatility in this notebook."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "8jpY8yCKWMyY"
},
"source": [
"# Account Performance"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "nzFrdhj6WMyb",
"outputId": "6d038ab6-6b7d-47db-b6e2-9922fee2a365"
},
"outputs": [],
"source": [
"# install packages\n",
"# !pip install pandas numpy\n",
"!pip install numerapi"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"id": "iKL3MID5WMyc"
},
"outputs": [],
"source": [
"# Import dependencies\n",
"import pandas as pd\n",
"import numpy as np\n",
"import numerapi"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def get_signals_resolve_status(api):\n",
" em_scores = [x for x in api.daily_submissions_performances(\"integration_test\")\n",
" if x['date'] is not None]\n",
" max_round = max(x['roundNumber'] for x in em_scores)\n",
" min_round = min(x['roundNumber'] for x in em_scores)\n",
" current_round_resolved = len([x for x in em_scores if x['roundNumber'] == max_round]) == 4\n",
" d = {\n",
" 'number': [],\n",
" 'resolvedGeneral': [],\n",
" }\n",
" for round_number in range(max_round, min_round - 1, -1):\n",
" d['number'].append(round_number)\n",
" if round_number == max_round:\n",
" d['resolvedGeneral'].append(current_round_resolved)\n",
" else:\n",
" d['resolvedGeneral'].append(True)\n",
" return pd.DataFrame(d).set_index(\"number\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"api = numerapi.SignalsAPI()\n",
"# Get round resolved states\n",
"resolve_status = get_signals_resolve_status(api)[\n",
" \"resolvedGeneral\"\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"id": "sAWcfJhHWMyd"
},
"outputs": [],
"source": [
"def get_compound_stake(daily_scores, resolved_rounds, weights):\n",
" \"\"\"\n",
" Starts with stake of 1 and compound weekly based on resolved rounds.\n",
" Signals tournament compounds every week.\n",
" \"\"\"\n",
" stake = pd.Series(data=1, index=daily_scores.index)\n",
" stake = stake.multiply(\n",
" 1\n",
" + weights[0] * resolved_rounds[\"correlation\"]\n",
" + weights[1] * resolved_rounds[\"mmc\"],\n",
" fill_value=1,\n",
" ).cumprod()\n",
"\n",
" return stake"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"id": "naC-QeoTWMye"
},
"outputs": [],
"source": [
"def get_account(account_name):\n",
" \"\"\"\n",
" Retrieve daily scores and calculate hypotethical 'unrealized brokerage\n",
" account value' for an account\n",
" \"\"\"\n",
" daily_performances = pd.DataFrame(api.daily_submissions_performances(account_name))\n",
"\n",
" # Each day, each active round submission promises a payout seperate from the others,\n",
" # therefore sum() of all rounds scores for a day are used instead of mean()\n",
" daily_scores = (\n",
" daily_performances.dropna(subset=[\"mmc\",\"correlation\"])\n",
" .sort_values(by=\"date\")\n",
" .groupby(\"date\")\n",
" .sum()[[\"correlation\", \"mmc\"]]\n",
" )\n",
"\n",
" # Last known date for each round submission\n",
" round_lastday = (\n",
" daily_performances.dropna()\n",
" .sort_values(by=\"date\")\n",
" .groupby(\"roundNumber\")\n",
" .last()\n",
" .reset_index()\n",
" .set_index(\"date\")\n",
" )\n",
"\n",
" # Pick resolved rounds\n",
" round_lastday[\"resolved\"] = round_lastday[\"roundNumber\"].map(resolve_status)\n",
" resolved_rounds = round_lastday[round_lastday.resolved]\n",
"\n",
" # Build the hypothethical stake that grows with each resolved round\n",
" money = pd.DataFrame(index=daily_scores.index)\n",
" money[\"stake_corr\"] = get_compound_stake(daily_scores, resolved_rounds, (2, 0))\n",
" money[\"stake_halfmmc\"] = get_compound_stake(daily_scores, resolved_rounds, (2, 0.5))\n",
" money[\"stake_1mmc\"] = get_compound_stake(daily_scores, resolved_rounds, (2, 1))\n",
" money[\"stake_2mmc\"] = get_compound_stake(daily_scores, resolved_rounds, (2, 2))\n",
" money[\"stake_3mmc\"] = get_compound_stake(daily_scores, resolved_rounds, (2, 3))\n",
"\n",
" # Daily scores are akin to daily changes in your brokerage account\n",
" money[\"score_corr\"] = daily_scores.correlation * 2.\n",
" money[\"score_halfmmc\"] = daily_scores.correlation * 2. + 0.5 * daily_scores.mmc\n",
" money[\"score_1mmc\"] = daily_scores.correlation * 2. + 1. * daily_scores.mmc\n",
" money[\"score_2mmc\"] = daily_scores.correlation * 2. + 2. * daily_scores.mmc\n",
" money[\"score_3mmc\"] = daily_scores.correlation * 2. + 3. * daily_scores.mmc\n",
"\n",
" # The paper value of your account changes with the daily score applied to the stake\n",
" money[\"account_corr\"] = money[\"stake_corr\"] * (1 + money[\"score_corr\"])\n",
" money[\"account_halfmmc\"] = money[\"stake_halfmmc\"] * (1 + money[\"score_halfmmc\"])\n",
" money[\"account_1mmc\"] = money[\"stake_1mmc\"] * (1 + money[\"score_1mmc\"])\n",
" money[\"account_2mmc\"] = money[\"stake_2mmc\"] * (1 + money[\"score_2mmc\"])\n",
" money[\"account_3mmc\"] = money[\"stake_3mmc\"] * (1 + money[\"score_3mmc\"])\n",
"\n",
" # All this work was to get our daily 'brokerage account value' equivalent\n",
" money = money[[\"account_corr\", \"account_halfmmc\", \"account_1mmc\", \"account_2mmc\", \"account_3mmc\"]]\n",
"\n",
" account_returns = pd.concat(\n",
" [\n",
" daily_scores,\n",
" money,\n",
" money.pct_change().add_prefix(\"pct_\"),\n",
" # (np.log(money) - np.log(money.shift(1))).add_prefix(\"log_\"),\n",
" ],\n",
" axis=1,\n",
" ).dropna()\n",
"\n",
" account_returns.name = account_name\n",
"\n",
" return account_returns"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"id": "KeUANHDgWMyf"
},
"outputs": [],
"source": [
"def get_performance(account_returns, lookback_days):\n",
" \"\"\"\n",
" Calculate annualized Sharpe and Sortino ratios\n",
" \"\"\"\n",
" # Extract the log return columns of interest\n",
" pct_returns = account_returns[\n",
" [column for column in account_returns.columns if column.startswith(\"pct_\")]\n",
" ].tail(lookback_days)\n",
" pct_returns.columns = [\"corr\", \"halfmmc\", \"1mmc\", \"2mmc\", \"3mmc\"]\n",
"\n",
" # Financial industry practice is 12 x monthly mean return\n",
" # for daily returns, use 252 trading days per year of US stock markets\n",
" annual_return = 252 * pct_returns.mean()\n",
"\n",
" # could use monthly compounding if you feel like bending those boundaries\n",
" # annual_return = np.power(1 + 21 * log_returns.mean(), 12) - 1\n",
"\n",
" # Annual volatility is calculated by multiplying daily volatility with sqrt of time \n",
" sharpe_ratio = annual_return / (np.sqrt(252) * pct_returns.std())\n",
"\n",
" # Sortino ratio considers stdev of negative days, positive suprises are a blessing\n",
" sortino_ratio = annual_return / (np.sqrt(252) * pct_returns[pct_returns < 0].std())\n",
"\n",
" # Let's see how long this track record is really for\n",
" actual_days = pd.Series({\"rows\": len(pct_returns)})\n",
"\n",
" return annual_return.add_prefix(\"annual_return_\").append(\n",
" sharpe_ratio.add_prefix(\"sharpe_\")\n",
" .append(sortino_ratio.add_prefix(\"sortino_\"))\n",
" .append(actual_days)\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"id": "GVyjoFHEWMyg"
},
"outputs": [],
"source": [
"# Default lookback is 6 months\n",
"DEFAULT_LOOKBACK = 130\n",
"\n",
"# helper function to hit the API and compute performances in a single call\n",
"def get_account_performance(account_name, lookback_days=DEFAULT_LOOKBACK):\n",
" account_returns = get_account(account_name)\n",
" return get_performance(account_returns, lookback_days).rename(account_name)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "JvH77NmzWMyh",
"outputId": "6448ed70-620b-463e-bd9e-b43a83c0ab9e"
},
"outputs": [
{
"data": {
"text/plain": [
"annual_return_corr 0.173975\n",
"annual_return_halfmmc 0.235767\n",
"annual_return_1mmc 0.308155\n",
"annual_return_2mmc 0.484636\n",
"annual_return_3mmc 0.703395\n",
"sharpe_corr 0.301550\n",
"sharpe_halfmmc 0.346659\n",
"sharpe_1mmc 0.393367\n",
"sharpe_2mmc 0.489467\n",
"sharpe_3mmc 0.587465\n",
"sortino_corr 0.466923\n",
"sortino_halfmmc 0.539508\n",
"sortino_1mmc 0.615599\n",
"sortino_2mmc 0.775143\n",
"sortino_3mmc 0.944447\n",
"rows 130.000000\n",
"Name: integration_test, dtype: float64"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# lets see how the integration_test is doing\n",
"get_account_performance(\"integration_test\")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"id": "IHYMfcpUWMyi"
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"6\n"
]
}
],
"source": [
"# your models and others you want to benchmark\n",
"account_list = [\n",
" \"nomi\",\n",
" \"metamike\",\n",
" # Others' accounts\n",
" \"uuazed2\",\n",
" \"leverage\",\n",
" \"dhi\",\n",
" \"degerhan_sb2\",\n",
"]\n",
"print(len(account_list))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"id": "GNB8JdTmWMyi"
},
"outputs": [],
"source": [
"# Rather than use get_account_performance helper,\n",
"# lets hit the API once for all accounts, which takes a few seconds\n",
"account_returns = [get_account(account) for account in account_list]"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/",
"height": 314
},
"id": "6-PxdWqlWMyi",
"outputId": "25cd4c39-5f8e-4d82-ae0c-add7b5b71c08"
},
"outputs": [
{
"data": {
"text/html": [
"<div>\n",
"<style scoped>\n",
" .dataframe tbody tr th:only-of-type {\n",
" vertical-align: middle;\n",
" }\n",
"\n",
" .dataframe tbody tr th {\n",
" vertical-align: top;\n",
" }\n",
"\n",
" .dataframe thead th {\n",
" text-align: right;\n",
" }\n",
"</style>\n",
"<table border=\"1\" class=\"dataframe\">\n",
" <thead>\n",
" <tr style=\"text-align: right;\">\n",
" <th></th>\n",
" <th>annual_return_corr</th>\n",
" <th>annual_return_halfmmc</th>\n",
" <th>annual_return_1mmc</th>\n",
" <th>annual_return_2mmc</th>\n",
" <th>annual_return_3mmc</th>\n",
" <th>sharpe_corr</th>\n",
" <th>sharpe_halfmmc</th>\n",
" <th>sharpe_1mmc</th>\n",
" <th>sharpe_2mmc</th>\n",
" <th>sharpe_3mmc</th>\n",
" <th>sortino_corr</th>\n",
" <th>sortino_halfmmc</th>\n",
" <th>sortino_1mmc</th>\n",
" <th>sortino_2mmc</th>\n",
" <th>sortino_3mmc</th>\n",
" <th>rows</th>\n",
" </tr>\n",
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>metamike</th>\n",
" <td>0.533970</td>\n",
" <td>0.696774</td>\n",
" <td>0.879171</td>\n",
" <td>1.302174</td>\n",
" <td>1.802434</td>\n",
" <td>0.699005</td>\n",
" <td>0.767600</td>\n",
" <td>0.836083</td>\n",
" <td>0.972063</td>\n",
" <td>1.106310</td>\n",
" <td>1.417809</td>\n",
" <td>1.571591</td>\n",
" <td>1.729212</td>\n",
" <td>2.054621</td>\n",
" <td>2.392869</td>\n",
" <td>67.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>nomi</th>\n",
" <td>0.464542</td>\n",
" <td>0.586919</td>\n",
" <td>0.720854</td>\n",
" <td>1.023069</td>\n",
" <td>1.370842</td>\n",
" <td>0.779970</td>\n",
" <td>0.829685</td>\n",
" <td>0.880107</td>\n",
" <td>0.981550</td>\n",
" <td>1.082599</td>\n",
" <td>1.202230</td>\n",
" <td>1.288201</td>\n",
" <td>1.376799</td>\n",
" <td>1.559763</td>\n",
" <td>1.748910</td>\n",
" <td>67.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>degerhan_sb2</th>\n",
" <td>0.362447</td>\n",
" <td>0.496732</td>\n",
" <td>0.651506</td>\n",
" <td>1.021685</td>\n",
" <td>1.471837</td>\n",
" <td>0.447987</td>\n",
" <td>0.517747</td>\n",
" <td>0.587129</td>\n",
" <td>0.724530</td>\n",
" <td>0.860060</td>\n",
" <td>0.734199</td>\n",
" <td>0.857371</td>\n",
" <td>0.981996</td>\n",
" <td>1.235381</td>\n",
" <td>1.494368</td>\n",
" <td>71.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>leverage</th>\n",
" <td>0.124795</td>\n",
" <td>0.233487</td>\n",
" <td>0.370309</td>\n",
" <td>0.730269</td>\n",
" <td>1.209949</td>\n",
" <td>0.141362</td>\n",
" <td>0.222191</td>\n",
" <td>0.303591</td>\n",
" <td>0.467599</td>\n",
" <td>0.632832</td>\n",
" <td>0.197890</td>\n",
" <td>0.314667</td>\n",
" <td>0.435285</td>\n",
" <td>0.688472</td>\n",
" <td>0.958957</td>\n",
" <td>67.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>dhi</th>\n",
" <td>-0.043286</td>\n",
" <td>-0.022913</td>\n",
" <td>0.007612</td>\n",
" <td>0.098763</td>\n",
" <td>0.229630</td>\n",
" <td>-0.078469</td>\n",
" <td>-0.034897</td>\n",
" <td>0.009997</td>\n",
" <td>0.101720</td>\n",
" <td>0.194547</td>\n",
" <td>-0.122814</td>\n",
" <td>-0.054868</td>\n",
" <td>0.015796</td>\n",
" <td>0.162427</td>\n",
" <td>0.315643</td>\n",
" <td>111.0</td>\n",
" </tr>\n",
" <tr>\n",
" <th>uuazed2</th>\n",
" <td>4.158188</td>\n",
" <td>17.517472</td>\n",
" <td>31.507589</td>\n",
" <td>164.856007</td>\n",
" <td>-5.822568</td>\n",
" <td>1.069932</td>\n",
" <td>1.672731</td>\n",
" <td>1.133086</td>\n",
" <td>1.275070</td>\n",
" <td>-0.807862</td>\n",
" <td>2.883473</td>\n",
" <td>8.554490</td>\n",
" <td>7.742938</td>\n",
" <td>13.921726</td>\n",
" <td>-0.782366</td>\n",
" <td>130.0</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
"</div>"
],
"text/plain": [
" annual_return_corr annual_return_halfmmc annual_return_1mmc \\\n",
"metamike 0.533970 0.696774 0.879171 \n",
"nomi 0.464542 0.586919 0.720854 \n",
"degerhan_sb2 0.362447 0.496732 0.651506 \n",
"leverage 0.124795 0.233487 0.370309 \n",
"dhi -0.043286 -0.022913 0.007612 \n",
"uuazed2 4.158188 17.517472 31.507589 \n",
"\n",
" annual_return_2mmc annual_return_3mmc sharpe_corr \\\n",
"metamike 1.302174 1.802434 0.699005 \n",
"nomi 1.023069 1.370842 0.779970 \n",
"degerhan_sb2 1.021685 1.471837 0.447987 \n",
"leverage 0.730269 1.209949 0.141362 \n",
"dhi 0.098763 0.229630 -0.078469 \n",
"uuazed2 164.856007 -5.822568 1.069932 \n",
"\n",
" sharpe_halfmmc sharpe_1mmc sharpe_2mmc sharpe_3mmc \\\n",
"metamike 0.767600 0.836083 0.972063 1.106310 \n",
"nomi 0.829685 0.880107 0.981550 1.082599 \n",
"degerhan_sb2 0.517747 0.587129 0.724530 0.860060 \n",
"leverage 0.222191 0.303591 0.467599 0.632832 \n",
"dhi -0.034897 0.009997 0.101720 0.194547 \n",
"uuazed2 1.672731 1.133086 1.275070 -0.807862 \n",
"\n",
" sortino_corr sortino_halfmmc sortino_1mmc sortino_2mmc \\\n",
"metamike 1.417809 1.571591 1.729212 2.054621 \n",
"nomi 1.202230 1.288201 1.376799 1.559763 \n",
"degerhan_sb2 0.734199 0.857371 0.981996 1.235381 \n",
"leverage 0.197890 0.314667 0.435285 0.688472 \n",
"dhi -0.122814 -0.054868 0.015796 0.162427 \n",
"uuazed2 2.883473 8.554490 7.742938 13.921726 \n",
"\n",
" sortino_3mmc rows \n",
"metamike 2.392869 67.0 \n",
"nomi 1.748910 67.0 \n",
"degerhan_sb2 1.494368 71.0 \n",
"leverage 0.958957 67.0 \n",
"dhi 0.315643 111.0 \n",
"uuazed2 -0.782366 130.0 "
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# then explore with lookback periods without hitting the API again\n",
"LOOKBACK = 130\n",
"MIN_ROWS = 30\n",
"\n",
"account_performances = pd.DataFrame(\n",
" [\n",
" get_performance(account_df, LOOKBACK).rename(account_df.name)\n",
" for account_df in account_returns\n",
" ]\n",
")\n",
"\n",
"account_performances[account_performances.rows > MIN_ROWS].sort_values(\n",
" by=\"sortino_3mmc\", ascending=False\n",
")"
]
}
],
"metadata": {
"colab": {
"collapsed_sections": [],
"include_colab_link": true,
"name": "live_sharpe_sortino.ipynb",
"provenance": []
},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.8"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment