I once worked on an EMV data preparation system, where I was required to carry out a load test on the system. To do this, I needed to test with lots of EMV cards.
But there was a problem.
Prior to now, developers and software testers tested with very few cards, just 1 to 10 cards. What I needed were lots of cards, about 1,000 to 10,000 cards, for this load test.
This brought about the need for a cards generator. A system that would generate, transform, and store cards ready for prep.
And here, I shall be discussing my journey to designing such system.
We had an existing system for cards creation, let's call it Alpha. Now, Alpha created cards by receiving a request with the details of a cardholder's user profile.
For this test, I had set up a user profile to work with.
In response, Alpha returns the created card data consisting of the PAN, CVV, Sequence Number, Track 2 data, etc., in JSON format.
If I wanted 5 cards, I needed make 5 calls via Postman to Alpha to create them for me. Meaning that if I needed 1,000 cards, I would have to make 1,000 calls.
A response from Alpha is delivered in JSON format, but the format required for EMV data preparation is in CSV format.
Hence the transformation of card data, from JSON to CSV.
This was a very fragile point, because each card data had to be represented in a strict position. Any card data that was placed in a wrong position would lead to data incorrectness which will result to a headache while preparing or printing the card.
My first idea was to design a system such that a user makes a call to generator API and gets a response in one synchronous call.
Interesting!
I went on to build it, but quickly hit an iceberg!
You see, there were many activities happening under the hood.
For instance, let's say I needed 10 cards, my generator would make 10 calls to Alpha, transform each card, persist into a file, and read out content of the file as an output.
This design worked well with little load, generating for 10 to 30 cards.
The challenge, however, was gateway timeouts.
As I increased the load, the upstream server took a long time to respond, and the connection broke.
A fix would have been to extend the timeout, but this wouldn't be scalable in terms of increasing load.
Say I increased the timeout seconds from 30ms to 1s, this may work when a user attempts to generate about 50 to 80 cards, but will not be feasible when generating 100 and more cards.
To solve this, I had to come up with a better design.
The second design focused on improving the system by making it an asynchronous process.
User sends a request to card generator and this request is loaded into DB as a job, then a job ID is returned as response to be used in the next call.
The first call kicks off a lambda function to start the processing, and the second call checks the status of the job.
The lambda function could run for 15 minutes.
Once processing is done, the user could download an output file.
At this point, users were able to request for 1,000 to 5,000 cards, get a job ID, and check the status of their request.
Fantastic!
Software testers and other members of my team now had a service to help in generating cards.
One evening, I decided to generate five of 10,000 cards so that I would have 50,000 cards.
By morning, only few hundreds were generated. The card generator had failed. There had been an intermittent network outage that night.
I felt bad because there was a pressing task I needed to close out which I couldn't, and now I had to figure out how to mitigate this.
Later that night, I read an article on bytebytego where I came across the word - retry mechanism.
After some research I came to understand that retry mechanisms were procedures put in place to automatically retry failed operations.
I decided to introduce a retry mechanism into the design.
Here, a worker would pick up the task of calling Alpha, transforming the response data and storing to file.
In an event of failure, the worker passes this task to a new worker through a queue and is terminated. The new worker retries the task after a configurable period and sets a retry count in database.
The system retries calling Alpha for a configured number of times before giving up.
I chose this approach so that a worker can be freed up after hitting a glitch, and also to handle decoupling.
This project was very interesting for me, as I designed, implemented and faced challenges that I needed to mitigate in real time.
Today, the system is very resilient and serves it's purpose.
However, I believe that the current design could still be improved. For instance, to speed up card generation, we could have multiple workers processing a large request in parallel, which will introduce an interesting kind of complexity.
Watch out for more of my articles as I share my experiences in systems design and building scalable, resilient applications.
Thanks for reading.