import multiprocessing | |
import time | |
from collections import deque | |
class ThrottleBarrier(): | |
def __init__( | |
self, | |
counter: multiprocessing.Value, | |
lock: multiprocessing.Lock, |
This logging setup configures Structlog to output pretty logs in development, and JSON log lines in production.
Then, you can use Structlog loggers or standard logging
loggers, and they both will be processed by the Structlog pipeline (see the hello()
endpoint for reference). That way any log generated by your dependencies will also be processed and enriched, even if they know nothing about Structlog!
Requests are assigned a correlation ID with the asgi-correlation-id
middleware (either captured from incoming request or generated on the fly).
All logs are linked to the correlation ID, and to the Datadog trace/span if instrumented.
This data "global to the request" is stored in context vars, and automatically added to all logs produced during the request thanks to Structlog.
You can add to these "global local variables" at any point in an endpoint with `structlog.contextvars.bind_contextvars(custom
This set of files is to help you set up a socket listener for Python logging in a production environment.
The other files are:
Filename | Purpose |
---|---|
prepare.sh | A Bash script to prepare the environment for testing. |
supervisor.conf | The Supervisor configuration file, which has entries for the listener and a multi-process web application. |
ensure_app.sh | A Bash script to ensure that Supervisor is running with the above configuration. |
log_listener.py | The socket listener program which receives log events and records them to a file. |
# --- my_functions.py | |
import pandas as pd | |
def avg_3wk_spend(spend: pd.Series) -> pd.Series: | |
"""Rolling 3 week average spend.""" | |
return spend.rolling(3).mean() | |
def spend_per_signup(spend: pd.Series, signups: pd.Series) -> pd.Series: |
Reproduced from https://thecodinginterface.com/blog/kafka-source-sink-with-apache-flink-table-api/
PyFlink is compatible with Python>=3.5<3.9
Process:
- Produce events and send to Kafka topic
- Set up streaming service via PyFlink DataStream API
- Read from Kafka source via PyFlink TABLE API
- Process data
❯ rm out.csv | |
❯ cat 1.py | |
from glob import glob | |
import mmap | |
files = glob("data/*") | |
files.sort(key=lambda x: int(x.split("/")[-1].split(".")[0])) | |
write_f = open("out.csv", "w+b") |
I liked the way Grokking the coding interview organized problems into learnable patterns. However, the course is expensive and the majority of the time the problems are copy-pasted from leetcode. As the explanations on leetcode are usually just as good, the course really boils down to being a glorified curated list of leetcode problems.
So below I made a list of leetcode problems that are as close to grokking problems as possible.
import logging | |
from multiprocessing import Pool, TimeoutError | |
logging.basicConfig(level=logging.INFO) | |
def timeout(func, args=None, kwds=None, timeout=10): | |
"""Call function and wait until timeout. |