Understanding and Ensuring Idempotency on Sensitive API Endpoints
A Fintech Perspective
What is idempotence?
Idempotence, in programming and mathematics, is a property of some operations, such that no matter how many times you execute them, you achieve the same result.
Elaborating on what is written above, idempotence ensures that the subsequent result of multiple iterations of an operation or function produces the same response as the first iteration, in other words, there are no side effects when the same operation is performed multiple times. Idempotency is applied at different levels of both programming and mathematics with their respective usage and behaviors.
Idempotency in REST APIs
This simply means that sending the same request multiple times will produce the same result, without changing the state of the server or the resource, guaranteeing uniformity.
In REST, some HTTP methods are idempotent by default, while some are not. The HTTP GET, PUT, and DELETE are examples of idempotent methods, to illustrate, consider a client makes a GET request to retrieve a resource with its ID, the result of this request will be the same for subsequent requests(same URI passed) for individual cases of when the resource actually exists and when it does not.
GET /item/:idAlso, consider a client performs a DELETE request to delete a resource from the server. The server processes the request, the resource gets deleted and the server returns 204(no content). Then the client repeats the same DELETE request and, as the resource has already been deleted, the server returns 404(not found).
DELETE /item/:idDespite the different status codes received by the client, the effect produced by a single DELETE request is the same effect of multiple DELETE requests to the same URI. Therefore, we can refer to those requests as idempotent. This is similar also applied to the GET example.
The POST request is not idempotent by default, by action, the client gets different responses after each subsequent repeated request is made on such endpoints. The POST method is mostly used for resource creation and many times the server is left to generate unique ids for these resources, this results in multiple(varied — new resource creation) side effects on the server, thereby resulting in equally varied responses sent back to the clients.
Ensuring idempotency on POST requests.
As I wrote earlier, idempotency is applied in different aspects of programming and many organizations use it to achieve their various business logic.
In the Fintech Space, you will find it very common amongst sensitive endpoints that need to cater for duplicates. For instance, for a particular POST endpoint that performs a money transfer between two users, idempotency is applied to ensure that when a request is sent out multiple times mistakenly by the same user, the system is smart enough to identify them as duplicates and acts accordingly.
The result of this operation is dependent upon the organization's logic, some might want to acknowledge or ignore it, while some might flag it as duplicate and return back to the user. In both cases, idempotency will be met as long as the desired behavior is not defeated(a side effect on the server).
Typically, for organizations that generate reference numbers/ids as identifiers(an idempotency-key) from the client’s end for their requests, they implement an interceptor of some sort on the server that performs a check in a cache to confirm if the reference numbers/ids(as keys) exist and then mark as a duplicate or return an idempotent response, for some other organizations(no client-side generated reference numbers/ids), certain deterministic fields are taken from the request body and are hashed together to form a key(the idempotency-key) that is used to perform a similar check as above to identify duplicates. In both cases, the determinant for a duplicate request/transaction from multiple identical requests will depend on the time to live, the expiration time of the items, or the nonexistence of the key in the cache(storage).
In cases where there are failed requests/transactions due to network failures or other complications, the idempotent-key is not added to the cache(store) and then subsequent retry mechanisms can be performed easily by the client. The idempotent-key is committed into the cache(store) after the first successful request is made.
Note that, the choice of storage to keep your idempotency-key is totally up to you, some people choose to use their databases and perform db operations for their checks.
Conclusion
Idempotency is one of many properties whose absence is felt more keenly when it first starts to stream at scale or if it starts to be used in “mission-critical” operations for an organization.
However, there are other ways organizations ensure and achieve idempotency on their systems, in the end it depends on how uniformity is guaranteed.
You can get more information about idempotency from the below links:
Below is also a blog article from Stripe about how they design predictable APIs with idempotency, it also gives you a wider perspective about the use cases of idempotency.
Hope you find this read interesting.
Cheers
