Saswata Dutta saswata-dutta

bs = hashlib.md5("Apple".encode('utf-8')).digest()
# b'\x9fb\x90\xf4CnZ#Q\xf1.\x03\xb6C<<'

int.from_bytes(bs[:4], 'big', signed=True)
# -1620930316

How to get a stack trace for each thread in a running Python script

Sometimes a Python script will simply hang forever with no indication of what is going wrong. Perhaps it's polling a service that will never return a value that allows the program to move forward.

Here's a way to see where the program is currently stuck, using pyrasite a tool for injecting code into running Python processes.

Install gdb and pyrasite

Install gdb.

	config:
	target: "http://localhost:8000"
	http:
	timeout: 10 # Responses have to be sent within 10 seconds or the request will be aborted
	processor: "./processor.js"

	phases:
	# Create 100 virtual users every second for 60 seconds
	- duration: 60 # seconds
	arrivalRate: 100 # virtual users

	aws logs filter-log-events \
	--log-group-name '/aws/lambda/my-lambda' \
	--log-stream-name-prefix '2022/07/31/[$LATEST]' \
	--start-time 1659247200000 \
	--end-time 1659254400000 \
	--filter-pattern '"EVENT ::"'

	# [--starting-token <value>]


	val oid_info = Seq((1,1), (1, 2), (1, 1), (2, 1), (2, 3), (3, 4), (3, 3), (3, 4)).toDF("oid", "ip")

	"""
	+---+---+
	\|oid\| ip\|
	+---+---+
	\| 1\| 1\|
	\| 1\| 2\|
	\| 1\| 1\|


	import spark.implicits._
	import org.apache.spark.sql.expressions.Window
	import org.apache.spark.sql.functions._
	import org.apache.spark.sql.types.IntegerType


	import org.apache.spark.sql.internal.SQLConf.SHUFFLE_PARTITIONS

	// need to set the shuffle partitions correctly based on data sizes and distribution

	"""
	Simple worker showing different worker ids sharing/incrementing the same memory region
	uwsgi --http :9090 --wsgi-file foobar.py --master --processes 4 --threads 2 --stats 127.0.0.1:9191 --sharedarea 2
	Then just keep refreshing localhost:9090
	"""

	import uwsgi

	INT_ORDS = {48, 49, 50, 51, 52, 53, 54, 55, 56, 57}

	filter @type = "REPORT"
	\| fields @maxMemoryUsed/1000000 as MemUsedMb
	\| stats max(MemUsedMb) by bin(1d)

	create table scores(id smallint primary key, score smallint not null);

	CREATE INDEX score_idx ON scores (score);


	WITH RECURSIVE
	cnt(x) AS (
	SELECT 1
	UNION ALL
	SELECT x+1 FROM cnt