Solution: "Exceeded 4 calls per second for api client" Python Shopify API - shopify_python_api

Readme.md

Solution for API call limit "shopify_python_api"

If you are using Python Shopify API and getting following error
"Exceeded 4 calls per second for api client. Reduce request rates to resume uninterrupted service."
but want your script to continue working with some timeout after that,
you can use following script from shopify_limits_patch.py.

For that just copy shopify_limits_patch.py to your project and import shopify_limits_patch.

Or if you want to call it implicitly import it, remove last line patch_shopify_with_limits()
and call it before all your shopify calls.

---

This is solution for issue described here: Shopify/shopify_python_api#256

If you have any suggestions for code - leave comment :)

shopify_limits_patch.py

import time

import pyactiveresource.connection
from shopify.base import ShopifyConnection


def patch_shopify_with_limits():
    func = ShopifyConnection._open

    def patched_open(self, *args, **kwargs):
        while True:
            try:
                return func(self, *args, **kwargs)

            except pyactiveresource.connection.ClientError as e:
                if e.response.code == 429:
                    retry_after = float(e.response.headers.get('Retry-After', 4))
                    print('Service exceeds Shopify API call limit, '
                          'will retry to send request in %s seconds' % retry_after)
                    time.sleep(retry_after)
                else:
                    raise e

    ShopifyConnection._open = patched_open


patch_shopify_with_limits()

validation.py

import os
from threading import Thread

import shopify

import shopify_limits_patch


def foo():
    for _ in range(15):
        shopify.Variant.find()

def main():
    # Add your Shopify URL here: "https://<shop_name>.myshopify.com/admin"
    shopify.ShopifyResource.set_site(os.environ.get('SHOPIFY_URL'))  

    for i in range(15):
        t = Thread(target=foo)
        t.start()

if __name__ == '__main__':
    main()

Thanks a ton for this @wowkin2
I was also able to handle Shopify server 5XX errors with this approach.

@coler-j I used this with django, just try not to import it inside any view. Preferably call this function after importing shopify.
Example code snippet -

import shopify
import pyactiveresource.connection
from shopify.base import ShopifyConnection


# For handling Shopify 429 - https://gist.github.com/wowkin2/079844c867a1a06ce15ea1e4ffdee87c
def patch_shopify_with_limits():
    func = ShopifyConnection._open

    def patched_open(self, *args, **kwargs):
        count = 0
        while True:
            count += 1
            try:
                return func(self, *args, **kwargs)

            except pyactiveresource.connection.ClientError as e:
                if e.response.code == 429:
                    # Checks for Retry-After header in 429 else defaults to 4 sec
                    retry_after = float(e.response.headers.get('Retry-After', 4))
                    log.info(f'Service exceeds Shopify API call limit, '
                             f'will retry to send request in {retry_after} seconds')
                    time.sleep(retry_after)
                else:
                    raise e
    ShopifyConnection._open = patched_open


patch_shopify_with_limits()

@api_view(http_method_names=['post', ])

I tried to implicitly import it, removed last line patch_shopify_with_limits() and called it before all my shopify calls. But I got errors for several endpoints. Seems like the recursion was using all memory. My django app will die after running for 1 - 2 days. After that I gracefully kill gunicorn and the app will be up again. Currently, I am trying to replace the while loop with a for loop running for several times. Just not sure if it can fix the issue yet.

Internal Server Error: /app_spo/getOptimizedDetail

RecursionError at /app_spo/getOptimizedDetail
maximum recursion depth exceeded while calling a Python object

It indicates that this line gives the error:
File "/var/www/app.seo-product-optimizer.com/app_spo/views/listProductAjax/shopify_limits_patch.py" in patched_open
13. return func(self, *args, **kwargs)

Request Method: GET
Django Version: 1.11.26
Python Version: 3.7.5

update:
Turn out that I am sending a bunch of ajax requests to my server. When there are too many requests, it will use up the memory and give the error.

works like a charm, thanks a lot.

Is there a way to do the same thing in requests library. I am using python requests to POST data to shopify (add new image) endpoint.

new_image = json.loads(util.shopify_post(f"https://{shop_url}/admin/api/2021-04/products/{product_id}/images.json" , {
                    "image": {
                        "attachment": optimized_image,
                        "filename":product_format+'.jpg',
                    }
                }).text)

where, the above code is getting called inside a for loop.

Once i hit the shopify benchmark for requests (40 per bucket) . it throws a error saying "Exceeded 2 calls per second for api client" and everything crashes.

Is there anything that can be done over here ?

i tried by putting sleep(3) seconds for every image being create inside the loop. it works . but don't think this is scalable enough and fast.

Any help would be highly appreaciated.

@shriDeveloper check for status_code and do retry or use some python queue. And that's it :)

@wowkin2 thanks for advice. i did some try with ratelimit library but didn't help too. retry seems good for now.

if reponse.status_code == 429:
       time.sleep(3)
       #shopify call again .

I tried to implicitly import it, removed last line patch_shopify_with_limits() and called it before all my shopify calls. But I got errors for several endpoints. Seems like the recursion was using all memory. My django app will die after running for 1 - 2 days. After that I gracefully kill gunicorn and the app will be up again. Currently, I am trying to replace the while loop with a for loop running for several times. Just not sure if it can fix the issue yet.

Internal Server Error: /app_spo/getOptimizedDetail

RecursionError at /app_spo/getOptimizedDetail maximum recursion depth exceeded while calling a Python object

It indicates that this line gives the error: File "/var/www/app.seo-product-optimizer.com/app_spo/views/listProductAjax/shopify_limits_patch.py" in patched_open 13. return func(self, *args, **kwargs)

Request Method: GET Django Version: 1.11.26 Python Version: 3.7.5

update: Turn out that I am sending a bunch of ajax requests to my server. When there are too many requests, it will use up the memory and give the error.

When you call patch_shopify_with_limits much times you get this errors, try replace to this code

import pyactiveresource.connection
from shopify.base import ShopifyConnection


_shopify_open = ShopifyConnection._open


def patch_shopify_with_limits():
    def patched_open(self, *args, **kwargs):
        while True:
            try:
                return _shopify_open(self, *args, **kwargs)
            except pyactiveresource.connection.ClientError as e:
                if e.response.code == 429:
                    retry_after = float(e.response.headers.get('Retry-After', 4))
                    print('Service exceeds Shopify API call limit, '
                          'will retry to send request in %s seconds' % retry_after)
                    time.sleep(retry_after)
                else:
                    raise e

    ShopifyConnection._open = patched_open


patch_shopify_with_limits()

Brilliant! Thanks!

Still working in 2024!

It works perfectly in late 2024, the only change i made is

retry_after = float(e.response.headers.get('retry-after', 4))

I created a shopify_patched module, and inside the init file, i added

from .shopify_limits_patch import shopify

To import the library, i replaced all:

import shopify

with

from app_api.api_modules.shopify_patched import shopify

Is that correct?

@b4p3p yeah, this way of import also works. Probably even better than proposed in 2018 😅
Can't check if 'retry-after' should be lower cased or any other. But people that use it - can easily do that.