miraculixx

Celery worker blocks on rate limited task
=========================================
by github.com/miraculixx

Problem:

    If a worker has a rate_limit active on some task, and that task
    arrives (is received)  more often than the rate limit interval, all
    worker processes will block on these task instances and stop
    consuming other tasks as soon as the prefetch count has maxed out

miraculixx

# no blank lines!
.gist
.gistignore
.git/*
.idea/*
__pycache_/*

miraculixx

version: '3'
services:
    rabbitmq:
        image: rabbitmq:latest
        ports:
          # 5672 amqp
          # 15672 mgmt ui
          # 1883 mqtt
          - "5672:5672"
          - "15672:15672"

miraculixx

# put in $HOME/.bash_k8s.rc
# . $HOME/bash_k8s.rc

# podlog <any pod name substring>
function podlog() {
   pod=`kubectl get pods --all-namespaces | grep " $1" | tr -s ' ' | cut -d ' ' -f 2 | head -n1`
   ns=`kubectl get pods --all-namespaces | grep " $1" | tr -s ' ' | cut -d ' ' -f 1 | head -n1`
   shift 1
   kubectl logs --namespace $ns $pod $*
}

miraculixx

These commands are based on a askubuntu answer http://askubuntu.com/a/581497 and https://askubuntu.com/questions/26498/choose-gcc-and-g-version
To install gcc-7 (gcc-7.2.0), I had to do more stuff as shown below.
USE THOSE COMMANDS AT YOUR OWN RISK. I SHALL NOT BE RESPONSIBLE FOR ANYTHING.
ABSOLUTELY NO WARRANTY.

If you are still reading let's carry on with the code.

sudo apt-get update && \
sudo apt-get install build-essential software-properties-common -y && \
sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y && \

miraculixx

#!/usr/bin/env bash
# adopted from https://raw.githubusercontent.com/X1011/git-directory-deploy/master/deploy.sh
# 
# changes 
# * deploy from current directory is possible
# * set options from either .deploy file or parameters 
set -o errexit #abort if any command fails
 
PREFIX=.
CONFIG=.deploy

miraculixx

from dateutil.parser import parse 

with open('data.csv') as fin:
    for line in csv.reader(fin):
       date1 = parse(row[1])
       date2 = datetime.now() - timedelta(days=3)
       if date1 > date2:
          continue
       process(row) # whatever your logic

miraculixx

import pandas as pd

text = """
Men      super men        size       Energy (J)    type    num      g
----------------------------------------------------------------------
50          1             1          1.0234E+03    A      abcd   12.1
20          7             4          5.0211E+02    A2 C   agcd   14.1
10          2             3         -1.0347E+02    B2     abkd   72.1
"""

miraculixx

from sodapy import Socrata

class SocrataPager(Socrata):
    def get(self, dataset, *args, limit=None, offset=0, where=None, order=None, page_size=None, **kwargs):
        """ get dataset by pages or within limit
        
        uses Socrata.get to page through results until limit is reached. This adds the page_size parameter
        which is passed as the limit kwarg to Socrata.get. limit is the total number of records to 
        return, across all pages. Note that the method is a generator.
        

miraculixx

How to run this

(output as of September 29, 2023)

$ python scraper.py
Does flying slower actually save fuel?
Is non-consented video recording admissable evidence in a civil trial in Maryland?
Iteration counts of AMG solver changes in parallel

Two switch flyback converter MOSFETs voltage stress