Idempotency keys: make API retries safe (so nobody is charged twice)

A customer taps "Pay". The request reaches your server, you charge the card, but the network drops the response. The app sees a timeout and retries. Without protection, the customer is charged twice. This is not rare. Timeouts, mobile networks and impatient users make duplicate requests normal. The standard fix is an idempotency key. The code and outputs below were run with Python 3 before publishing.

What idempotent means

An operation is idempotent if doing it many times has the same effect as doing it once. GET, PUT and DELETE are meant to be idempotent in HTTP. POST, which creates things, is not, so we have to add the protection ourselves.

The idea

1. The client generates a unique idempotency key (usually a random UUID) for one intended action and sends it in a header, e.g. Idempotency-Key: 8f1c....
2. The server remembers the key together with the response it produced.
3. If the same key arrives again, the server does not do the work again. It returns the saved response.
4. Retries reuse the same key. A genuinely new payment uses a new key.

Several payment providers use this pattern in their public APIs, and an IETF draft describes an Idempotency-Key header, so it is a familiar convention.

A working example

Save as payments.py:

import hashlib
import json
import sqlite3


def make_db():
    db = sqlite3.connect(":memory:")
    db.executescript("""
        CREATE TABLE payments (id INTEGER PRIMARY KEY, user TEXT, amount INTEGER);
        CREATE TABLE idempotency (
            key TEXT PRIMARY KEY,
            request_hash TEXT NOT NULL,
            status INTEGER NOT NULL,
            response TEXT NOT NULL
        );
    """)
    return db


def fingerprint(body):
    return hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()


def create_payment(db, body, idempotency_key=None):
    """Return (status, response). The same key with the same body never charges twice."""
    request_hash = fingerprint(body)
    if idempotency_key:
        row = db.execute("SELECT request_hash, status, response FROM idempotency WHERE key = ?", (idempotency_key,)).fetchone()
        if row:
            if row[0] != request_hash:
                return 422, {"error": "Idempotency key reused with a different request"}
            return row[1], json.loads(row[2])  # replay the stored answer

    try:
        with db:  # payment and the saved key are written in ONE transaction
            cur = db.execute("INSERT INTO payments (user, amount) VALUES (?, ?)", (body["user"], body["amount"]))
            response = {"payment_id": cur.lastrowid, "amount": body["amount"]}
            if idempotency_key:
                db.execute("INSERT INTO idempotency VALUES (?, ?, 201, ?)", (idempotency_key, request_hash, json.dumps(response)))
        return 201, response
    except sqlite3.IntegrityError:  # two requests with the same key raced; the loser replays
        row = db.execute("SELECT status, response FROM idempotency WHERE key = ?", (idempotency_key,)).fetchone()
        return row[0], json.loads(row[1])


if __name__ == "__main__":
    db = make_db()
    body = {"user": "alice", "amount": 500}
    print("1st request, key abc:", create_payment(db, body, "abc"))
    print("retry, same key     :", create_payment(db, body, "abc"))
    print("same key, other body:", create_payment(db, {"user": "alice", "amount": 900}, "abc"))
    print("no key, twice       :", create_payment(db, body), create_payment(db, body))
    print("payments in database:", db.execute("SELECT count(*) FROM payments").fetchone()[0])

Output of python3 payments.py:

1st request, key abc: (201, {'payment_id': 1, 'amount': 500})
retry, same key     : (201, {'payment_id': 1, 'amount': 500})
same key, other body: (422, {'error': 'Idempotency key reused with a different request'})
no key, twice       : (201, {'payment_id': 2, 'amount': 500}) (201, {'payment_id': 3, 'amount': 500})
payments in database: 3

Reading the result

• First request creates payment 1.
• Retry with the same key returns the same answer, and no second payment is created.
• Same key, different body is rejected with 422. Reusing a key for a different request is a client bug, and silently replaying the old answer would hide it.
• No key gives no protection: two requests made two payments. Making the key required for risky endpoints is a stricter choice.

The details that matter

One transaction. The payment and the saved key are written together. If the server crashed between them, you could have a payment with no key (a retry would charge again) or a key with no payment (a retry would say "done" for something that never happened). The last test proves that a failure leaves no half-saved key.

A unique constraint is the real lock. key is a PRIMARY KEY. If two identical requests arrive at the same instant, both may pass the "do we know this key?" check, but only one insert can succeed. The loser gets an IntegrityError and replays the winner's response. Checking first and inserting later, without a unique constraint, has a race condition.

Fingerprint the request. We store a hash of the body and compare it. Same key plus a different body is an error, not a replay.

Store the response. Return exactly what the first call returned, including the status code, so the client cannot tell the difference.

Scope keys. In a real system, scope the key to the user or account, so one customer's key cannot collide with or reveal another's.

Expire keys. Keep them for a limited time (for example 24 hours or a few days) and delete old rows. Retries happen within minutes or hours, not months.

Tests

Save as test_payments.py:

import unittest

import payments


class IdempotencyTest(unittest.TestCase):
    def setUp(self):
        self.db = payments.make_db()
        self.body = {"user": "bob", "amount": 100}

    def count(self):
        return self.db.execute("SELECT count(*) FROM payments").fetchone()[0]

    def test_retry_with_same_key_charges_once(self):
        first = payments.create_payment(self.db, self.body, "k1")
        second = payments.create_payment(self.db, self.body, "k1")
        self.assertEqual(first, second)
        self.assertEqual(self.count(), 1)

    def test_different_keys_are_different_payments(self):
        payments.create_payment(self.db, self.body, "k1")
        payments.create_payment(self.db, self.body, "k2")
        self.assertEqual(self.count(), 2)

    def test_key_reuse_with_other_body_is_rejected(self):
        payments.create_payment(self.db, self.body, "k1")
        status, _ = payments.create_payment(self.db, {"user": "bob", "amount": 999}, "k1")
        self.assertEqual(status, 422)
        self.assertEqual(self.count(), 1)

    def test_failure_leaves_no_half_saved_key(self):
        with self.assertRaises(KeyError):
            payments.create_payment(self.db, {"user": "bob"}, "k9")  # amount missing
        self.assertEqual(self.db.execute("SELECT count(*) FROM idempotency").fetchone()[0], 0)


if __name__ == "__main__":
    unittest.main()

$ python3 -m unittest -v
Ran 4 tests in 0.009s

OK

What about requests still in progress?

This simple version writes the result in the same transaction as the work, so there is no "in progress" state to see. When the work calls an external service (a card processor, an email provider), you cannot put it in your database transaction. A common design then:

1. Insert the key with status processing.
2. Do the external call, passing your key along if the provider supports idempotency keys.
3. Save the result and mark the key done.
4. If a retry arrives while processing, return 409 Conflict (or ask the client to wait) rather than starting a second call.

This is harder than it looks. A crash after the external call but before saving needs recovery logic or reconciliation. The example above does not cover it.

Other places this pattern helps

• Webhooks and queue consumers: the same message can be delivered more than once, so processing must be idempotent. Track processed event ids.
• Background jobs that may be retried.
• Order, booking and signup endpoints.
• Non-HTTP approaches: a natural unique constraint, such as one order per cart id, can also prevent duplicates.

Common mistakes

• Generating a new key on every retry, which defeats the purpose. The client must reuse the key for the same action.
• Checking for the key without a unique constraint, which allows races.
• Saving the key before the work finishes and never cleaning up after a failure.
• Making keys guessable or shared across users.
• Using idempotency keys as a replacement for authentication and authorisation. They do not replace them.

Limits of this example

It is a teaching sketch. The IntegrityError handler assumes the error came from the key. A real system should check which constraint failed and handle other errors separately. It also has no expiry job and no user scoping, and the in-memory database disappears on exit.

Takeaway: assume every request can arrive twice. Give create-type endpoints an idempotency key, store the response with the key in the same transaction as the work, rely on a unique constraint to win races, and reject key reuse with a different body.

---

Backend series, Part 3 (reliable APIs): 1 of 5.