Pavel Kirilin s3rius

Here's a small comparison between taskiq, celery and dramatiq.

This benchmark is a modified version of an original dramatiq benchmark.

To run the benchmark locally:

docker compose up -d --wait      # To startup services used for testing.
pip install -r requirements.txt  # To install python dependencies
python runner.py --retries 10 -t 10 -t 100 -t 1_000 -t 10_000

To run the benchmark install:

pip install scyllapy acsylla uvloop

And then run it with scylla ip address as the first positional argument. For example:

python benches.py "172.29.0.2" --uvloop

Here's an example of dynamic scheduler for taskiq. It uses postgres to store all tasks.

We created custom schedule source that is capable of storing and retrieving scheduled tasks.

Start workers by running:

taskiq worker tkq:broker --fs-discover

This router manages to operate with multiple brokers. The idea is pretty straightforward, you define which broker runs on which node, they use the same url, but different queues.

Running default worker command will not work, since it wouldn't be able to decide which broker to run. To surpass this issue, we can define custom receiver class, which picks a broker based on node parameter.

Now insted of

taskiq worker a:router

	FROM python:3.12-slim-bookworm

	WORKDIR /app

	RUN pip install aiokafka

	COPY main.py /app/main.py

	CMD ["python", "/app/main.py"]

	[package]
	name = "itertest"
	version = "0.1.0"
	edition = "2021"

	[dependencies]
	futures = "0.3.28"
	pyo3 = "0.19.2"
	pyo3-asyncio = { version = "0.19.0", features = ["tokio-runtime"] }
	tokio = { version = "1.32.0", features = ["sync"] }