kemingy

import os
import time
from concurrent import futures
import signal


class Model:
    def __init__(self):
        self.pid = os.getpid()

kemingy

import msgpack
import httpx


if __name__ == "__main__":
    url = 'http://localhost:8000/'
    with httpx.Client() as client:
        client.headers.update({'Content-Type': 'application/msgpack'})
        packer = msgpack.Packer(
            autoreset=True,

kemingy

package main

import (
	"log"
	"net"
	"time"
)


func main() {

kemingy

Environments

• CPU: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
• GPU: NVIDIA V100
• Memory: 251GiB
• OS: Ubuntu 16.04.6 LTS (Xenial Xerus)

Docker Images:

• tensorflow/tensorflow:latest-gpu
• tensorflow/serving:latest-gpu

kemingy

Install

pip install cython

Build

• setuptools
• setup.py

kemingy

from werkzeug.datastructures import MultiDict
from starlette.datastructures import ImmutableMultiDict
from falcon.util.uri import parse_query_string
from pydantic import BaseModel, root_validator
from typing import List
from typing_extensions import Literal


class Query(BaseModel):
    cmd: List[Literal['ls', 'rm', 'cd']]

kemingy

Ref:

• https://konghq.com/blog/how-to-design-a-scalable-rate-limiting-algorithm/
• https://www.figma.com/blog/an-alternative-approach-to-rate-limiting/
• https://engineering.classdojo.com/blog/2015/02/06/rolling-rate-limiter/

Rate Limiting Algorithms

Leaky Bucket

Just a queue. If the queue is full, then additional requests will be discarded.

kemingy

Using Python as Glue Summary

https://docs.scipy.org/doc/numpy/user/c-info.python-as-glue.html

tools	f2py	Cython	ctypes	SWIG
source lang	Fortran	C/C++	C	C/C++
pros	It allows clean separation of Python with compiled code while still allowing for separate distribution of the extension module.	Easily and quickly work with multidimensional arrays. Easily to distributed.	Clean separation of C code from Python Code. Easy integration with numpy. Full argument checking.	It can (almost) parse the headers and auto produce the interface.
cons	It requires the existence of a Fortran compiler in order for a user to install the code.	Need to be familiar with C. Can be hard to reuse the C code for other non-Python-related projects.	Hard to distributed. Need shared-libraries for the code. Very little support for C++.	It must be done using the concept of typemaps.

kemingy

Descriptor

Define: a descriptor is an object attribute with "binding behavior", one whose attribute access has been overridden by methods in the descriptor protocal.

a.x -> a.__dict__['x'] -> type(a).__dict__['x']

Descriptor Protocol

• descr.__get__(self, obj, type=None) -> value
• descr.__set__(self, obj, value) -> None

kemingy

Chinese Word Segmentation

Ref:

• https://arxiv.org/ftp/arxiv/papers/1901/1901.06079.pdf

Tagging

• B: begin
• M: middle
• E: end