tejaslodaya
/ apply_df_by_multiprocessing.py
Created
February 3, 2017 09:12
pandas DataFrame apply multiprocessing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import multiprocessing | |
| import pandas as pd | |
| import numpy as np | |
| def _apply_df(args): | |
| df, func, num, kwargs = args | |
| return num, df.apply(func, **kwargs) | |
| def apply_by_multiprocessing(df,func,**kwargs): | |
| workers=kwargs.pop('workers') |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| performance (netflix) - https://qconsp.com/system/files/keynotes-slides/qcon_sp_keynote_-_martin_spier.pdf | |
| - metricas impressionantes deles (requests, logs, latencia, instancias etc) | |
| - arquitetura (caches nos provedores) | |
| - 3 regioes aws | |
| - ec2 reserved (se nao tá streaming, usa pra recomendacoes, encoding etc) | |
| - build: amis ubuntu | |
| - deploy: 4000 por dia | |
| - tudo stateless | |
| - ferramentas (captura de metricas da redhat, flamegraph, flamescope etc) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| -Xmx2048m | |
| -Xss256k | |
| -XX:+UseG1GC | |
| -XX:InitiatingHeapOccupancyPercent=65 | |
| -XX:G1HeapRegionSize=16m | |
| -XX:MaxGCPauseMillis=100 |
OlderNewer