Created
September 22, 2023 05:03
-
-
Save rohanpm/5d7bc4cefb772d8cfb34597b336e028c to your computer and use it in GitHub Desktop.
Force exodus-gw to retry every S3 request at least three times (even if it succeeded)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/exodus_gw/aws/client.py b/exodus_gw/aws/client.py | |
| index 5cbf1f5..beb52f5 100644 | |
| --- a/exodus_gw/aws/client.py | |
| +++ b/exodus_gw/aws/client.py | |
| @@ -1,3 +1,4 @@ | |
| +import logging | |
| import os | |
| import aioboto3 | |
| @@ -38,12 +39,21 @@ class S3ClientWrapper: | |
| client.meta.events.register( | |
| "needs-retry.s3.CreateMultipartUpload", self.no_redirects | |
| ) | |
| + client.meta.events.register("needs-retry.s3", self.force_retry) | |
| return client | |
| async def __aexit__(self, exc_type, exc, tb): | |
| await self._client_context.__aexit__(exc_type, exc, tb) | |
| + @staticmethod | |
| + def force_retry(**kwargs): | |
| + log = logging.getLogger("exodus-gw") | |
| + log.debug("needs-retry.s3 invoked: %s", kwargs) | |
| + if kwargs["attempts"] < 3: | |
| + log.warning("Forcing an S3 retry!") | |
| + return True | |
| + | |
| @staticmethod | |
| def no_redirects(**kwargs): | |
| # An event handler for needs-s3.retry.* events which will disable implicit |
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This can be used to trigger the UnseekableStreamError condition locally.
What it does is force every S3 request to be retried at least 3 times, even if the request actually succeeded. This is good enough to simulate the case where a request really does have to be retried due to some failure, and hits the UnseekableStreamError.
If testing this with exodus-rsync, you MUST use a unique file for every test, because the "retry" happens after the S3 request actually succeeded - which means the content made it into S3 successfully and a follow-up test will detect that the object is already present, and won't upload it.
Example of testing with this:
With corresponding server logs: