Samuel Sampaio samukasmk

Simple queues reprocessing

Executing

python simple_queues.py

Example

Proof of Concept: Implementing MongoEngine lib with support for multi database connections

Reason:

Currently MongoEngine library only support a single database connection, managing collections by Document classes in global scope.

It has a support to change database connections by switch_db but it's made by a global class attribute called ._meta['db_alias']

When you define many sub ReferenceFields in a hierarchical tree, only first Document object has its ._meta['db_alias'] atttibute changed, not working to query Documents that implement other documents.

Installing

pip install confluent-kafka

Initialization

It's important to declare producer and consumer objects in global scope for better performance

This is a easy example of how to profiling memory usage by memory_profiler library in 5 steps:

1.) Install memory_profiler libraries

pip install -U memory_profiler matplotlib

	import time

	#
	# Creating two large sets for testing
	#

	a = set(range(1, 1000000)) # Set with 1 million elements
	b = set(range(500000, 1500000)) # Set that partially overlaps with set 'a'
	print()

	# reference: https://docs.gspread.org/en/latest/user-guide.html

	# Install the required libraries
	!pip install gspread google-auth

	# Import libraries
	import gspread
	from google.colab import auth
	from oauth2client.client import GoogleCredentials
	from google.auth import default

	import asyncio

	async def create_async_results(total_items: int):
	for r in range(total_items):
	await asyncio.sleep(1)
	yield r

	async def example_of_async_for_loop():
	async for item in create_async_results(10):
	print(item)

	import asyncio
	from random import randint


	async def execute_your_task(task_number: int):
	""" Specific task to run in parallel """

	# display begin of parallel task in console
	print('-> starting parallel task:', task_number)

	import asyncio
	from random import randint


	async def execute_your_task(task_number: int):
	""" Specific task to run in parallel """

	# display begin of parallel task in console
	print('-> starting parallel task:', task_number)