Skip to content

Instantly share code, notes, and snippets.

@balouf

balouf/profiling.ipynb

Created May 5, 2021 12:21
Show Gist options
  • Save balouf/ffa61afdf765bce275f9feced79428aa to your computer and use it in GitHub Desktop.
Save balouf/ffa61afdf765bce275f9feced79428aa to your computer and use it in GitHub Desktop.
Download ZIP
Introduction to Python Profiling
Display the source blob
Display the rendered blob
Raw
profiling.ipynb
{
"cells": [
{
"metadata": {
"slideshow": {
"slide_type": "skip"
},
"ExecuteTime": {
"start_time": "2021-05-05T08:10:27.967407Z",
"end_time": "2021-05-05T08:10:27.981929Z"
},
"trusted": true
},
"cell_type": "code",
"source": "from IPython.core.display import display, HTML\ndisplay(HTML(\"\"\"<style>\n.prompt_container { display: none !important; }\n.prompt { display: none !important; }\n.run_this_cell { display: none !important; }\n</style>\"\"\"))",
"execution_count": 1,
"outputs": [
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "<style>\n.prompt_container { display: none !important; }\n.prompt { display: none !important; }\n.run_this_cell { display: none !important; }\n</style>"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"cell_type": "markdown",
"source": "# Python Profiling"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# How to make your code fast?"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Think of something efficient from an algorithmic point of view\n- Write a correct code\n- Use code acceleration techniques (numpy, numba, cython, parallelization, ...)"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# When is profiling required"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Choose amongst possible options\n- Find out what needs to be optimized"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Pros and cons of profiling"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Avoids spending time on parts that DO NOT need optimization\n- The speed gain can sometimes turn a code easy to read/maintain into a black box\n- Profiling and code acceleration are sometimes tricky (e.g. numba)"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Note: for this session, we will not use any hardcore optimization technique (numba etc...)"
},
{
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"cell_type": "markdown",
"source": "## Time it, the basic solution"
},
{
"metadata": {
"cell_style": "center"
},
"cell_type": "markdown",
"source": "Time it runs a code multiple times and provide basic statistics. \n\nUsage: `%timeit mycode()`"
},
{
"metadata": {
"cell_style": "center"
},
"cell_type": "markdown",
"source": "- Pros:\n - Simple, out-of-the box on Jupyter, no overhead\n - Adapted to comparison\n- Cons:\n - No info beyond the execution time\n - Not adapted to optimization"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Example: Fibonacci"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:11:02.576010Z",
"end_time": "2021-05-05T09:11:02.590109Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "def fibo_simple(n):\n if n < 3:\n return 1\n else:\n return fibo_simple(n-1)+fibo_simple(n-2)",
"execution_count": 2,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:11:08.672888Z",
"end_time": "2021-05-05T09:11:24.012079Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "print(fibo_simple(30))\n%timeit fibo_simple(30)",
"execution_count": 3,
"outputs": [
{
"output_type": "stream",
"text": "832040\n186 ms ± 1.58 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Explicit dynamic programming"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:14:25.475423Z",
"end_time": "2021-05-05T09:14:25.484720Z"
},
"trusted": true,
"cell_style": "center"
},
"cell_type": "code",
"source": "def fibo_dp(n, results=None):\n if results is None:\n results = dict()\n if n in results:\n return results[n]\n if n < 3:\n return results.setdefault(n, 1)\n else:\n return results.setdefault(n, fibo_dp(n-1, results) + fibo_dp(n-2, results)) ",
"execution_count": 4,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:14:27.213939Z",
"end_time": "2021-05-05T09:14:39.163475Z"
},
"trusted": true,
"cell_style": "center"
},
"cell_type": "code",
"source": "print(fibo_dp(30))\n%timeit fibo_dp(30)",
"execution_count": 5,
"outputs": [
{
"output_type": "stream",
"text": "832040\n14.7 µs ± 254 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Iterative variant"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:15:50.593409Z",
"end_time": "2021-05-05T09:15:50.605700Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "def fibo_derec(n):\n if n < 3:\n return 1\n else:\n r1 = 1\n r2 = 1\n for i in range(n-2):\n r = r1 + r2\n r2 = r1\n r1 = r\n return r",
"execution_count": 6,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:15:57.792955Z",
"end_time": "2021-05-05T09:16:12.319517Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "print(fibo_derec(30))\n%timeit fibo_derec(30) ",
"execution_count": 7,
"outputs": [
{
"output_type": "stream",
"text": "832040\n1.79 µs ± 52.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Direct computation"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:16:58.154575Z",
"end_time": "2021-05-05T09:16:58.363358Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "import numpy as np\nϕ1 = (1+np.sqrt(5))/2\nϕ2 = (1-np.sqrt(5))/2\na = (1-ϕ2)/(ϕ1-ϕ2)\nb = (ϕ1-1)/(ϕ1-ϕ2)\ndef fibo_al(n):\n n=n-1\n return int(a*ϕ1**n+b*ϕ2**n)",
"execution_count": 8,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:17:01.776191Z",
"end_time": "2021-05-05T09:17:11.981009Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "print(fibo_al(30))\n%timeit fibo_al(30)",
"execution_count": 9,
"outputs": [
{
"output_type": "stream",
"text": "832040\n1.25 µs ± 35.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "fragment"
},
"ExecuteTime": {
"start_time": "2021-05-05T09:18:16.305443Z",
"end_time": "2021-05-05T09:18:16.310657Z"
},
"trusted": true
},
"cell_type": "code",
"source": "print(f\"{fibo_derec(100)} vs {fibo_al(100)}\")",
"execution_count": 10,
"outputs": [
{
"output_type": "stream",
"text": "354224848179261915075 vs 354224848179263111168\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Automatic dynamic programming"
},
{
"metadata": {
"cell_style": "split",
"ExecuteTime": {
"start_time": "2021-05-05T10:11:14.735612Z",
"end_time": "2021-05-05T10:11:14.742267Z"
},
"trusted": true
},
"cell_type": "code",
"source": "# Reminder\ndef fibo_simple(n):\n if n < 3:\n return 1\n else:\n return fibo_simple(n-1)+fibo_simple(n-2)",
"execution_count": 57,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:11:15.996022Z",
"end_time": "2021-05-05T10:11:16.008545Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "from functools import lru_cache\n@lru_cache(maxsize=None)\ndef fibo_cached(n):\n if n < 3:\n return 1\n else:\n return fibo_cached(n-1)+fibo_cached(n-2)",
"execution_count": 58,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:13:18.457653Z",
"end_time": "2021-05-05T10:13:18.671202Z"
},
"trusted": true
},
"cell_type": "code",
"source": "print(fibo_cached(30))\n%timeit -n 1 -r 100000 fibo_cached(30)",
"execution_count": 67,
"outputs": [
{
"output_type": "stream",
"text": "832040\nThe slowest run took 3193.00 times longer than the fastest. This could mean that an intermediate result is being cached.\n250 ns ± 1.32 µs per loop (mean ± std. dev. of 100000 runs, 1 loop each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"trusted": true
},
"cell_type": "code",
"source": "",
"execution_count": null,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Did we trick timeit with lru_cache?"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:13:21.996452Z",
"end_time": "2021-05-05T10:13:22.003532Z"
},
"trusted": true
},
"cell_type": "code",
"source": "fibo_cached.cache_info()",
"execution_count": 68,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 68,
"data": {
"text/plain": "CacheInfo(hits=91430245, misses=30, maxsize=None, currsize=30)"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:28:34.101299Z",
"end_time": "2021-05-05T09:28:34.110040Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "def fibo_limited_cache(n):\n fibo_cached.cache_clear()\n return fibo_cached(n)",
"execution_count": 14,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:28:38.320782Z",
"end_time": "2021-05-05T09:28:52.600376Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "print(fibo_limited_cache(30))\n%timeit fibo_limited_cache(30)",
"execution_count": 15,
"outputs": [
{
"output_type": "stream",
"text": "832040\n17.5 µs ± 404 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:29:18.220238Z",
"end_time": "2021-05-05T09:29:18.227833Z"
},
"trusted": true
},
"cell_type": "code",
"source": "fibo_cached.cache_info()",
"execution_count": 16,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 16,
"data": {
"text/plain": "CacheInfo(hits=27, misses=30, maxsize=None, currsize=30)"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Did we trick timeit with lru_cache?"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- In practice, a function that you optimize is often called multiple times on runtime, so the gain of `fibo_cache` may be real!\n\n- Like all caching techniques, beware of memory consumption.\n\n- In 3.9, a `@cache` has been introduced (faster, no size limit)"
},
{
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"cell_type": "markdown",
"source": "## cProfile, raw diving"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "cProfile analyzes the calls.\n\nUsage: `cProfile.run('mycode()', sort = 'cumtime')`"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Pros:\n - Out-of-the box\n - Pinpoints the bottleneck functions\n - Adapted when you have lots of simple functions\n- Cons:\n - Needs to decipher the output\n - Not adapted when bottleneck lies in loops/tests/allocations"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Example: Age of Eve"
},
{
"metadata": {
"cell_style": "split"
},
"cell_type": "markdown",
"source": "Simple model for the mitochondrial Eve: there is a first generation of $ n $ individuals (the potential eves). At each next generation, there are $ n $ individuals as well. The parent of each individual is chosen uniformly i.i.d. among the individuals of the previous generation. The game stops when one individual of the first generation is the unique ancestor of the current generation: Eve has been found."
},
{
"metadata": {
"cell_style": "split"
},
"cell_type": "markdown",
"source": "<img src='https://upload.wikimedia.org/wikipedia/commons/9/92/MtDNA-MRCA-generations-Evolution.svg'>"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Best algorithmic approach"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Starts from today: current generation has *n* non-extinct families.\n- For each non-extinct family, draw mother among the $n$ individuals of previous generation\n- Count the distinct mothers; this is the number of non-extinct families of previous generation\n- Iterate until the number of non-extinct families is 1\n- Result is the number of iterations"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# First solution"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:39:35.698589Z",
"end_time": "2021-05-05T09:39:35.708742Z"
},
"trusted": true
},
"cell_type": "code",
"source": "def age_of_eve(n = 20000):\n families = n\n k = 0\n while not(families == 1):\n mothers = [np.random.randint(n) for _ in range(families)]\n families = len(np.unique(mothers))\n k+=1\n return k",
"execution_count": 18,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:39:39.908612Z",
"end_time": "2021-05-05T09:39:40.106402Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "age_of_eve(2000)",
"execution_count": 21,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 21,
"data": {
"text/plain": "2241"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:39:45.919077Z",
"end_time": "2021-05-05T09:40:07.030083Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "%timeit age_of_eve()",
"execution_count": 22,
"outputs": [
{
"output_type": "stream",
"text": "2.66 s ± 355 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Find out the bottlenecks"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:41:17.698561Z",
"end_time": "2021-05-05T09:41:20.048435Z"
},
"trusted": true
},
"cell_type": "code",
"source": "import cProfile\ncProfile.run('age_of_eve()', sort = 'cumtime')",
"execution_count": 23,
"outputs": [
{
"output_type": "stream",
"text": " 910076 function calls in 2.331 seconds\n\n Ordered by: cumulative time\n\n ncalls tottime percall cumtime percall filename:lineno(function)\n 1 0.000 0.000 2.331 2.331 {built-in method builtins.exec}\n 1 0.000 0.000 2.331 2.331 <string>:1(<module>)\n 1 0.088 0.088 2.331 2.331 <ipython-input-18-389deea09e67>:1(age_of_eve)\n 32385 0.134 0.000 1.545 0.000 <ipython-input-18-389deea09e67>:5(<listcomp>)\n 391912 1.411 0.000 1.411 0.000 {method 'randint' of 'numpy.random.mtrand.RandomState' objects}\n 32385 0.030 0.000 0.693 0.000 <__array_function__ internals>:2(unique)\n 32385 0.040 0.000 0.657 0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}\n 32385 0.054 0.000 0.617 0.000 arraysetops.py:138(unique)\n 32385 0.224 0.000 0.422 0.000 arraysetops.py:310(_unique1d)\n 64770 0.024 0.000 0.146 0.000 _asarray.py:110(asanyarray)\n 64770 0.122 0.000 0.122 0.000 {built-in method numpy.array}\n 32385 0.103 0.000 0.103 0.000 {method 'sort' of 'numpy.ndarray' objects}\n 32385 0.034 0.000 0.034 0.000 {built-in method numpy.empty}\n 32385 0.033 0.000 0.033 0.000 {method 'flatten' of 'numpy.ndarray' objects}\n 32385 0.016 0.000 0.021 0.000 arraysetops.py:125(_unpack_tuple)\n 64770 0.010 0.000 0.010 0.000 {built-in method builtins.len}\n 32385 0.006 0.000 0.006 0.000 arraysetops.py:133(_unique_dispatcher)\n 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n\n\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Removing the randint bottleneck"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "In pure Python, calls to the random generator can be more costly than the generation itself (this does not happen in Numba). Try to draw all mothers of a generation at once."
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:43:25.680464Z",
"end_time": "2021-05-05T09:43:25.697690Z"
},
"trusted": true
},
"cell_type": "code",
"source": "def age_of_eve_2(n = 20000):\n families = n\n k = 0\n while not(families == 1):\n mothers = np.random.randint(n, size = families)\n families = len(np.unique(mothers))\n k+=1\n return k",
"execution_count": 24,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:43:27.019326Z",
"end_time": "2021-05-05T09:43:38.453900Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%timeit age_of_eve_2()",
"execution_count": 25,
"outputs": [
{
"output_type": "stream",
"text": "The slowest run took 5.03 times longer than the fastest. This could mean that an intermediate result is being cached.\n1.42 s ± 822 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Removing the randint bottleneck"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:44:22.957067Z",
"end_time": "2021-05-05T09:44:24.592275Z"
},
"trusted": true
},
"cell_type": "code",
"source": "cProfile.run('age_of_eve_2()', sort = 'cumtime')",
"execution_count": 26,
"outputs": [
{
"output_type": "stream",
"text": " 1055354 function calls in 1.612 seconds\n\n Ordered by: cumulative time\n\n ncalls tottime percall cumtime percall filename:lineno(function)\n 1 0.000 0.000 1.612 1.612 {built-in method builtins.exec}\n 1 0.000 0.000 1.612 1.612 <string>:1(<module>)\n 1 0.073 0.073 1.612 1.612 <ipython-input-24-29981ca6bd4d>:1(age_of_eve_2)\n 84428 0.083 0.000 1.127 0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}\n 42214 0.312 0.000 0.793 0.000 {method 'randint' of 'numpy.random.mtrand.RandomState' objects}\n 42214 0.038 0.000 0.740 0.000 <__array_function__ internals>:2(unique)\n 42214 0.066 0.000 0.646 0.000 arraysetops.py:138(unique)\n 42214 0.281 0.000 0.515 0.000 arraysetops.py:310(_unique1d)\n 42214 0.038 0.000 0.481 0.000 <__array_function__ internals>:2(prod)\n 42214 0.057 0.000 0.398 0.000 fromnumeric.py:2912(prod)\n 42214 0.112 0.000 0.342 0.000 fromnumeric.py:70(_wrapreduction)\n 42214 0.163 0.000 0.163 0.000 {method 'reduce' of 'numpy.ufunc' objects}\n 42214 0.127 0.000 0.127 0.000 {method 'sort' of 'numpy.ndarray' objects}\n 84428 0.032 0.000 0.065 0.000 _asarray.py:110(asanyarray)\n 42214 0.039 0.000 0.039 0.000 {built-in method numpy.empty}\n 42214 0.039 0.000 0.039 0.000 {method 'flatten' of 'numpy.ndarray' objects}\n 42214 0.034 0.000 0.034 0.000 {built-in method builtins.getattr}\n 84428 0.033 0.000 0.033 0.000 {built-in method numpy.array}\n 42214 0.022 0.000 0.029 0.000 arraysetops.py:125(_unpack_tuple)\n 42214 0.026 0.000 0.026 0.000 fromnumeric.py:71(<dictcomp>)\n 84428 0.014 0.000 0.014 0.000 {built-in method builtins.len}\n 42214 0.009 0.000 0.009 0.000 arraysetops.py:133(_unique_dispatcher)\n 42214 0.009 0.000 0.009 0.000 fromnumeric.py:2907(_prod_dispatcher)\n 42214 0.008 0.000 0.008 0.000 {method 'items' of 'dict' objects}\n 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n\n\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Removing the randint bottleneck"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:45:42.371313Z",
"end_time": "2021-05-05T09:45:42.379317Z"
},
"trusted": true
},
"cell_type": "code",
"source": "def age_of_eve_3(n=20000):\n rand_vect = np.random.randint(n, size = n)\n rand_index = 0\n families = n\n k = 0\n while not(families == 1):\n if rand_index+families>n:\n rand_vect = np.random.randint(n, size = n)\n rand_index = 0\n mothers = rand_vect[rand_index:(rand_index+families)]\n families = len(np.unique(mothers))\n rand_index += families\n k += 1\n return k",
"execution_count": 27,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:45:43.568068Z",
"end_time": "2021-05-05T09:45:44.527118Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "age_of_eve_3()",
"execution_count": 28,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 28,
"data": {
"text/plain": "65580"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:45:46.031921Z",
"end_time": "2021-05-05T09:45:50.501588Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "%timeit age_of_eve_3()",
"execution_count": 29,
"outputs": [
{
"output_type": "stream",
"text": "553 ms ± 249 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Removing the randint bottleneck"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:45:59.199127Z",
"end_time": "2021-05-05T09:45:59.558306Z"
},
"trusted": true
},
"cell_type": "code",
"source": "cProfile.run('age_of_eve_3()', sort = 'cumtime')",
"execution_count": 30,
"outputs": [
{
"output_type": "stream",
"text": " 251034 function calls in 0.340 seconds\n\n Ordered by: cumulative time\n\n ncalls tottime percall cumtime percall filename:lineno(function)\n 1 0.000 0.000 0.340 0.340 {built-in method builtins.exec}\n 1 0.000 0.000 0.340 0.340 <string>:1(<module>)\n 1 0.028 0.028 0.340 0.340 <ipython-input-27-10a2e8b38047>:1(age_of_eve_3)\n 16722 0.015 0.000 0.298 0.000 <__array_function__ internals>:2(unique)\n 16742 0.019 0.000 0.281 0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}\n 16722 0.026 0.000 0.261 0.000 arraysetops.py:138(unique)\n 16722 0.108 0.000 0.212 0.000 arraysetops.py:310(_unique1d)\n 16722 0.059 0.000 0.059 0.000 {method 'sort' of 'numpy.ndarray' objects}\n 33444 0.012 0.000 0.026 0.000 _asarray.py:110(asanyarray)\n 16722 0.017 0.000 0.017 0.000 {built-in method numpy.empty}\n 16722 0.015 0.000 0.015 0.000 {method 'flatten' of 'numpy.ndarray' objects}\n 33444 0.014 0.000 0.014 0.000 {built-in method numpy.array}\n 20 0.011 0.001 0.012 0.001 {method 'randint' of 'numpy.random.mtrand.RandomState' objects}\n 16722 0.008 0.000 0.010 0.000 arraysetops.py:125(_unpack_tuple)\n 33444 0.004 0.000 0.004 0.000 {built-in method builtins.len}\n 16722 0.003 0.000 0.003 0.000 arraysetops.py:133(_unique_dispatcher)\n 20 0.000 0.000 0.001 0.000 <__array_function__ internals>:2(prod)\n 20 0.000 0.000 0.001 0.000 fromnumeric.py:2912(prod)\n 20 0.000 0.000 0.001 0.000 fromnumeric.py:70(_wrapreduction)\n 20 0.000 0.000 0.000 0.000 {method 'reduce' of 'numpy.ufunc' objects}\n 20 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}\n 20 0.000 0.000 0.000 0.000 fromnumeric.py:71(<dictcomp>)\n 20 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}\n 20 0.000 0.000 0.000 0.000 fromnumeric.py:2907(_prod_dispatcher)\n 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n\n\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Removing the np.unique bottleneck"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:47:06.211698Z",
"end_time": "2021-05-05T09:47:06.230134Z"
},
"trusted": true
},
"cell_type": "code",
"source": "def age_of_eve_4(n=20000):\n rand_vect = np.random.randint(n, size = n)\n rand_index = 0\n families = n\n k = 0\n while not(families == 1):\n if rand_index+families>n:\n rand_vect = np.random.randint(n, size = n)\n rand_index = 0\n families = len(set(rand_vect[rand_index:(rand_index+families)]))\n rand_index += families\n k += 1\n return k",
"execution_count": 31,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:47:07.373289Z",
"end_time": "2021-05-05T09:47:07.563950Z"
},
"trusted": true
},
"cell_type": "code",
"source": "age_of_eve_4()",
"execution_count": 32,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 32,
"data": {
"text/plain": "45130"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:47:09.059353Z",
"end_time": "2021-05-05T09:47:27.056809Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%timeit age_of_eve_4()",
"execution_count": 33,
"outputs": [
{
"output_type": "stream",
"text": "232 ms ± 31.1 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Removing the np.unique bottleneck"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:47:35.872815Z",
"end_time": "2021-05-05T09:47:36.024661Z"
},
"trusted": true
},
"cell_type": "code",
"source": "cProfile.run('age_of_eve_4()', sort = 'cumtime')",
"execution_count": 34,
"outputs": [
{
"output_type": "stream",
"text": " 29746 function calls in 0.132 seconds\n\n Ordered by: cumulative time\n\n ncalls tottime percall cumtime percall filename:lineno(function)\n 1 0.000 0.000 0.132 0.132 {built-in method builtins.exec}\n 1 0.000 0.000 0.132 0.132 <string>:1(<module>)\n 1 0.119 0.119 0.132 0.132 <ipython-input-31-095f0562598b>:1(age_of_eve_4)\n 20 0.010 0.000 0.011 0.001 {method 'randint' of 'numpy.random.mtrand.RandomState' objects}\n 29542 0.003 0.000 0.003 0.000 {built-in method builtins.len}\n 20 0.000 0.000 0.001 0.000 <__array_function__ internals>:2(prod)\n 20 0.000 0.000 0.001 0.000 {built-in method numpy.core._multiarray_umath.implement_array_function}\n 20 0.000 0.000 0.001 0.000 fromnumeric.py:2912(prod)\n 20 0.000 0.000 0.001 0.000 fromnumeric.py:70(_wrapreduction)\n 20 0.000 0.000 0.000 0.000 {method 'reduce' of 'numpy.ufunc' objects}\n 20 0.000 0.000 0.000 0.000 {built-in method builtins.getattr}\n 20 0.000 0.000 0.000 0.000 fromnumeric.py:71(<dictcomp>)\n 20 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects}\n 20 0.000 0.000 0.000 0.000 fromnumeric.py:2907(_prod_dispatcher)\n 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n\n\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "### Limits of method profiling "
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:49:37.913450Z",
"end_time": "2021-05-05T09:49:37.926844Z"
},
"trusted": true
},
"cell_type": "code",
"source": "import cProfile\ncProfile.run('fibo_derec(300)', sort = 'cumtime')",
"execution_count": 35,
"outputs": [
{
"output_type": "stream",
"text": " 4 function calls in 0.000 seconds\n\n Ordered by: cumulative time\n\n ncalls tottime percall cumtime percall filename:lineno(function)\n 1 0.000 0.000 0.000 0.000 {built-in method builtins.exec}\n 1 0.000 0.000 0.000 0.000 <string>:1(<module>)\n 1 0.000 0.000 0.000 0.000 <ipython-input-6-4c949ccf320b>:1(fibo_derec)\n 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}\n\n\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"cell_type": "markdown",
"source": "## snakeViz, cProfile with style"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "snakevis is a shiny frontend for cProfile. \n\nUsage: `%load_ext snakeviz` once, then `%snakeviz mycode()`"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Pros:\n - Beautiful and interactive\n - Bottlenecks are easy to spot\n- Cons:\n - Needs to be installed (`pip install snakeviz`)\n - Same info that raw cProfile, only heavier"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# snakeViz, cProfile with style"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:50:08.131641Z",
"end_time": "2021-05-05T09:50:14.874437Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%load_ext snakeviz\n%snakeviz age_of_eve()",
"execution_count": 36,
"outputs": [
{
"output_type": "stream",
"text": " \n*** Profile stats marshalled to file 'C:\\\\Users\\\\fabienma\\\\AppData\\\\Local\\\\Temp\\\\tmp5tmh8msk'. \nEmbedding SnakeViz in this document...\n",
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "\n<iframe id='snakeviz-48ccf1dc-ad87-11eb-b5e1-0cdd246eae91' frameborder=0 seamless width='100%' height='1000'></iframe>\n<script>document.getElementById(\"snakeviz-48ccf1dc-ad87-11eb-b5e1-0cdd246eae91\").setAttribute(\"src\", \"http://\" + document.location.hostname + \":8080/snakeviz/C%3A%5CUsers%5Cfabienma%5CAppData%5CLocal%5CTemp%5Ctmp5tmh8msk\")</script>\n"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# snakeViz, cProfile with style"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:51:30.306184Z",
"end_time": "2021-05-05T09:51:35.713848Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%snakeviz age_of_eve_2()",
"execution_count": 37,
"outputs": [
{
"output_type": "stream",
"text": " \n*** Profile stats marshalled to file 'C:\\\\Users\\\\fabienma\\\\AppData\\\\Local\\\\Temp\\\\tmpbsgi52ny'. \nEmbedding SnakeViz in this document...\n",
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "\n<iframe id='snakeviz-78fc2c86-ad87-11eb-a26c-0cdd246eae91' frameborder=0 seamless width='100%' height='1000'></iframe>\n<script>document.getElementById(\"snakeviz-78fc2c86-ad87-11eb-a26c-0cdd246eae91\").setAttribute(\"src\", \"http://\" + document.location.hostname + \":8080/snakeviz/C%3A%5CUsers%5Cfabienma%5CAppData%5CLocal%5CTemp%5Ctmpbsgi52ny\")</script>\n"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# snakeViz, cProfile with style"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:52:22.352683Z",
"end_time": "2021-05-05T09:52:27.461118Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%snakeviz age_of_eve_3()",
"execution_count": 38,
"outputs": [
{
"output_type": "stream",
"text": " \n*** Profile stats marshalled to file 'C:\\\\Users\\\\fabienma\\\\AppData\\\\Local\\\\Temp\\\\tmpjfgo_t0q'. \nEmbedding SnakeViz in this document...\n",
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "\n<iframe id='snakeviz-97d39e02-ad87-11eb-8d0e-0cdd246eae91' frameborder=0 seamless width='100%' height='1000'></iframe>\n<script>document.getElementById(\"snakeviz-97d39e02-ad87-11eb-8d0e-0cdd246eae91\").setAttribute(\"src\", \"http://\" + document.location.hostname + \":8080/snakeviz/C%3A%5CUsers%5Cfabienma%5CAppData%5CLocal%5CTemp%5Ctmpjfgo_t0q\")</script>\n"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# snakeViz, cProfile with style"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:52:40.920345Z",
"end_time": "2021-05-05T09:52:44.870989Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%snakeviz age_of_eve_4()",
"execution_count": 39,
"outputs": [
{
"output_type": "stream",
"text": " \n*** Profile stats marshalled to file 'C:\\\\Users\\\\fabienma\\\\AppData\\\\Local\\\\Temp\\\\tmpwpqcyt92'. \nEmbedding SnakeViz in this document...\n",
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "\n<iframe id='snakeviz-a2355064-ad87-11eb-8c4b-0cdd246eae91' frameborder=0 seamless width='100%' height='1000'></iframe>\n<script>document.getElementById(\"snakeviz-a2355064-ad87-11eb-8c4b-0cdd246eae91\").setAttribute(\"src\", \"http://\" + document.location.hostname + \":8080/snakeviz/C%3A%5CUsers%5Cfabienma%5CAppData%5CLocal%5CTemp%5Ctmpwpqcyt92\")</script>\n"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"cell_type": "markdown",
"source": "## Line profiler, when methods are not enough"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "Line profile analyzes... line by line! \n\n`%load_ext line_profiler`, then `%lprun -f functolook mycode()`"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Pros:\n - Bottlenecks are usually easier to spot than with raw cProfile\n - runtime bottlenecks (loop, allocations, tests...) can be found\n\n- Cons:\n - Needs to be installed (`pip install line_profiler`)\n - Cannot zoom inside a line (e.g. if you like big fat one-liners)\n - You may have to *follow the rabbit*"
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Line profiler, when methods are not enough"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:54:47.620859Z",
"end_time": "2021-05-05T09:54:51.802902Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%load_ext line_profiler\n%lprun -f age_of_eve age_of_eve()",
"execution_count": 40,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Line profiler, when methods are not enough"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:55:26.702935Z",
"end_time": "2021-05-05T09:55:27.616433Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%lprun -f age_of_eve_2 age_of_eve_2()",
"execution_count": 41,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Line profiler, when methods are not enough"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:55:45.153651Z",
"end_time": "2021-05-05T09:55:46.945701Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%lprun -f age_of_eve_3 age_of_eve_3()",
"execution_count": 42,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Line profiler, when methods are not enough"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:56:14.533136Z",
"end_time": "2021-05-05T09:56:14.887132Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%lprun -f age_of_eve_4 age_of_eve_4()",
"execution_count": 43,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Limits of profiling"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:57:16.787015Z",
"end_time": "2021-05-05T09:57:16.794629Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%lprun -f fibo_derec fibo_derec(300)",
"execution_count": 44,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Limits of profiling"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:58:03.875692Z",
"end_time": "2021-05-05T09:58:03.888237Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "def fibo_derec_2(n):\n if n < 3:\n return 1\n else:\n r1 = 1\n r2 = 1\n for i in range(n-2):\n r1, r2 = r1 + r2, r1\n return r1",
"execution_count": 45,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:58:05.315435Z",
"end_time": "2021-05-05T09:58:05.329676Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "fibo_derec_2(30)",
"execution_count": 46,
"outputs": [
{
"output_type": "execute_result",
"execution_count": 46,
"data": {
"text/plain": "832040"
},
"metadata": {}
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:58:26.163271Z",
"end_time": "2021-05-05T09:58:26.179043Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%lprun -f fibo_derec_2 fibo_derec_2(300)",
"execution_count": 47,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Limits of profiling"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:59:10.285625Z",
"end_time": "2021-05-05T09:59:12.269719Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "%timeit fibo_derec(300)",
"execution_count": 50,
"outputs": [
{
"output_type": "stream",
"text": "24.5 µs ± 1.04 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T09:59:13.255516Z",
"end_time": "2021-05-05T09:59:15.150380Z"
},
"trusted": true,
"cell_style": "split"
},
"cell_type": "code",
"source": "%timeit fibo_derec_2(300)",
"execution_count": 51,
"outputs": [
{
"output_type": "stream",
"text": "23.2 µs ± 416 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Limits of profiling"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:02:45.094327Z",
"end_time": "2021-05-05T10:02:47.196085Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "from numba import njit\n\n@njit\ndef fibo_derec_2(n):\n if n < 3:\n return 1\n else:\n r1 = 1\n r2 = 1\n for i in range(n-2):\n r1, r2 = r1 + r2, r1\n return r1",
"execution_count": 52,
"outputs": []
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:03:14.143193Z",
"end_time": "2021-05-05T10:03:17.068286Z"
},
"cell_style": "split",
"trusted": true
},
"cell_type": "code",
"source": "print(fibo_derec_2(30))\n%timeit fibo_derec_2(300)",
"execution_count": 54,
"outputs": [
{
"output_type": "stream",
"text": "832040\n363 ns ± 5.33 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)\n",
"name": "stdout"
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Limits of profiling"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:03:28.753529Z",
"end_time": "2021-05-05T10:03:32.727659Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%snakeviz fibo_derec_2(300)",
"execution_count": 55,
"outputs": [
{
"output_type": "stream",
"text": " \n*** Profile stats marshalled to file 'C:\\\\Users\\\\fabienma\\\\AppData\\\\Local\\\\Temp\\\\tmp7l0p6i6j'. \nEmbedding SnakeViz in this document...\n",
"name": "stdout"
},
{
"output_type": "display_data",
"data": {
"text/plain": "<IPython.core.display.HTML object>",
"text/html": "\n<iframe id='snakeviz-245a7f12-ad89-11eb-8ae1-0cdd246eae91' frameborder=0 seamless width='100%' height='1000'></iframe>\n<script>document.getElementById(\"snakeviz-245a7f12-ad89-11eb-8ae1-0cdd246eae91\").setAttribute(\"src\", \"http://\" + document.location.hostname + \":8080/snakeviz/C%3A%5CUsers%5Cfabienma%5CAppData%5CLocal%5CTemp%5Ctmp7l0p6i6j\")</script>\n"
},
"metadata": {}
}
]
},
{
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"cell_type": "markdown",
"source": "# Limits of profiling"
},
{
"metadata": {
"ExecuteTime": {
"start_time": "2021-05-05T10:03:37.466541Z",
"end_time": "2021-05-05T10:03:37.481125Z"
},
"trusted": true
},
"cell_type": "code",
"source": "%lprun -f fibo_derec_2 fibo_derec_2(300)",
"execution_count": 56,
"outputs": []
},
{
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"cell_type": "markdown",
"source": "# This is it!"
},
{
"metadata": {},
"cell_type": "markdown",
"source": "- Sources (among others)\n - https://jakevdp.github.io/PythonDataScienceHandbook/01.07-timing-and-profiling.html\n - https://toucantoco.com/en/tech-blog/tech/python-performance-optimization\n - Personal experience\n- Not covered in this presentation:\n - memory profiling (see `%mprun` of line_profiler)\n - py-spy (top for Python https://github.com/benfred/py-spy)"
}
],
"metadata": {
"kernelspec": {
"name": "python3",
"display_name": "Python 3",
"language": "python"
},
"toc": {
"nav_menu": {},
"number_sections": true,
"sideBar": true,
"skip_h1_title": true,
"base_numbering": 1,
"title_cell": "Table of Contents",
"title_sidebar": "Contents",
"toc_cell": false,
"toc_position": {},
"toc_section_display": true,
"toc_window_display": false
},
"language_info": {
"name": "python",
"version": "3.7.7",
"mimetype": "text/x-python",
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"pygments_lexer": "ipython3",
"nbconvert_exporter": "python",
"file_extension": ".py"
},
"celltoolbar": "Slideshow",
"gist": {
"id": "",
"data": {
"description": "Introduction to Python Profiling",
"public": true
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment