Skip to content

Instantly share code, notes, and snippets.

View simon-mo's full-sized avatar
:shipit:

Simon Mo simon-mo

:shipit:
893 followers · 2 following
View GitHub Profile
@simon-mo
simon-mo / cache_usage_2hr_01_20_2020.csv
Created January 20, 2020 23:38
ip count is_travis_ip
104.196.57.92 18199 True
35.227.97.188 15061 True
34.74.253.255 10694 True
35.196.72.151 8435 True
35.237.8.208 8208 True
35.196.158.85 7857 True
35.237.212.185 6797 True
52.179.196.222 5702 False
35.237.56.208 5599 True
@simon-mo
simon-mo / README.md
Created January 21, 2020 23:21 
pip install uvicorn beautifulsoup4 requests aiohttp lxml scrapy
  • compute.py pings the mock_server.py and figure out the views per second
  • mock_server.py can be started with uvicorn mock_server:app
  • scrapy_spider is the existing solution, run it with scrapy runspider scrapy_spider.py
  • sync_spider use requests and fire request synchrounously
  • async_spider use aiohttp and fire 4 request concurrently at a time
@simon-mo
simon-mo / image_preprocessing.py
Created January 29, 2020 23:19
from __future__ import print_function, absolute_import, division
import rpc
import os
import sys
import numpy as np
import torch
from torchvision import transforms
from torch.autograd import Variable
from PIL import Image, ImageEnhance
import logging
@simon-mo
simon-mo / 0.8.2.html
Last active February 20, 2020 22:10
<!DOCTYPE html>
<html>
<head>
<style>
.error {
color: red;
}
</style>
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm//vega@5"></script>
<script type="text/javascript" src="https://cdn.jsdelivr.net/npm//[email protected]"></script>
@simon-mo
simon-mo / 0.8.1.log
Created February 20, 2020 19:38
== Status ==
Memory usage on this node: 37.0/480.3 GiB
Using FIFO scheduling algorithm.
Resources requested: 0/640 CPUs, 0/8 GPUs, 0.0/2526.95 GiB heap, 0.0/128.42 GiB objects
Result logdir: /home/ubuntu/ray_results/atari-impala
Number of trials: 4 (4 TERMINATED)
+---------------------------------------------+------------+-------+-----------------------------+--------+------------------+-------------+----------+
| Trial name | status | loc | env | iter | total time (s) | timesteps | reward |
|---------------------------------------------+------------+-------+-----------------------------+--------+------------------+-------------+----------|
| IMPALA_BreakoutNoFrameskip-v4_511a1830 | TERMINATED | | BreakoutNoFrameskip-v4 | 39 | 699.024 | 3050000 | 44.21 |
@simon-mo
simon-mo / asyncio_ray.py
Last active February 26, 2020 21:10
import asyncio
import ray
ray.init()
@ray.remote
class AsyncWorker:
async def do_work(self):
print("Started")
await asyncio.sleep(0.5) # simulate network I/O
@simon-mo
simon-mo / asyncio_ray_concurrent_actor.py
Last active February 28, 2020 03:39
@ray.remote
class AsyncActor:
def __init__(self):
...
async def some_function(self):
...
await asyncio.sleep(1)
...
@simon-mo
simon-mo / asyncio_ray_async_get.py
Last active February 26, 2020 21:30
# ray futures are directly awaited.
result = await object_id_1
result = await task.remote()
# Use asyncio.wait to wait with timeout
done_futures, not_done_futures = await asyncio.wait([ObjecIDs...], timeout=...)
# Use asyncio.gather to wait for a batch
results = await asyncio.gather([object_id_1.future(), object_id_2.future()])
@simon-mo
simon-mo / asyncio_ray_load_balancer.py
Created February 26, 2020 21:15
@ray.remote
class LoadBalancer:
async def proxy_request(self, query):
actor = self.choose_actor()
return await actor.execute.remote(query)
@simon-mo
simon-mo / asyncio_ray_offload.py
Created February 26, 2020 21:18
# before
async def compute_intensive_coroutine():
do_work() # cpu heavy task here
... # blocking the event loop
... # threading doesn't help because of GIL
# after
async def compute_intensive_coroutine();
await do_work.remote()
# offload task to ray worker