On a real-money gaming platform, the worst bug is not the one that takes the site down: it is the one that credits the same deposit twice, or lets a bet go through for free. Neither shows up in the logs as an error. Here I explain how I structured the wallet of Girowin's platform so that this does not happen, and what is still on the way.
Where duplicates come from
- Game providers resend the same bet, win or cancellation when they do not get an answer in time.
- A deposit credit travels as a RabbitMQ RPC. If the reply is lost after the commit, the caller tries again.
- Two bets from the same player in the same millisecond read the same balance.
- Scheduled jobs run on more than one replica.
- A process can crash mid-flow: one service commits, the other never gets to record it.
Some of these really happened and shaped the design, like a credit that repeated on a retry because it ran before the idempotent record, and a deposit credit that failed silently.
The rule: idempotency in the database, not in discipline
Instead of trusting every handler to remember to check whether it already processed an event, I put the guarantee in a constraint. Every wallet movement has a unique key made of the wallet, the movement kind and the external id of whatever originated it: the deposit id, the provider transaction id, the bet id.
When a resend hits that key, the uniqueness error becomes an explicit duplicate error and the whole transaction rolls back, balance included. The entry is written with the same EntityManager that changed the balance, and the new balance comes from an UPDATE with RETURNING: the "before" is derived from it, never from an earlier read another transaction might have changed.
For deposit credits, a resend does not fail: the service returns the receipt of the original entry, with the balance before and after, and refuses if the user, brand or amount do not match.
The path of a deposit
The diagram shows the path end to end. Each step is idempotent on its own.
sequenceDiagram
participant G as Payment gateway
participant C as casino
participant Q as RabbitMQ
participant W as gaming
participant DB as PostgreSQL
G->>C: payment notification (through the queue)
C->>C: confirms the deposit (already confirmed = skip)
C->>C: in-process event deposit.confirmed
C->>Q: credit RPC keyed by the deposit id
Q->>W: credit
W->>DB: balance and entry in one transaction
alt resend of the same deposit
DB-->>W: unique key already exists
W-->>C: receipt of the original entry, nothing credited
else first time
W-->>C: balance before and after
end
C->>C: marks CREDITED or schedules a retry with backoff- Callback: the same status already applied is skipped, a confirmed deposit is never downgraded by a late status, and a late confirmation after expiration wins.
- In-process event: deposit.confirmed is only emitted on the transition to confirmed; if the deposit was already credited, non-idempotent effects such as promotions and affiliates are skipped.
- Credit: the service rereads the deposit from the database, skips it if already credited and sends the deposit id as the key; on failure it records the failure and schedules a retry, without throwing.
- Retry: the job uses the deposit id as its id, so there are never two retries of the same deposit; 5 attempts with exponential backoff from 30 s, and the failed job is kept. When attempts run out, the deposit becomes a case for a person to look at.
- Expiration: a conditional UPDATE only expires what is still pending, without overriding a concurrent confirmation.
Concurrent bets and withdrawals
A bet debit reads the account with FOR UPDATE and only then splits the amount between real and bonus balance. A withdrawal takes an advisory lock per withdrawal, locks the account and writes a debit receipt in the same transaction; a replay of the same withdrawal returns the original receipt after checking user, brand, amount and wallet. Every path follows the same lock order, so there are no deadlocks.
Money is not a float
The whole money path uses decimal.js, and amounts go into numeric(18,8) columns. In a bet debit, the effective amount is derived from the balance already rounded to 8 places, so wallet, bet and bonus agree exactly with what the database stores.
Checking the chain
Every entry stores the balance before and after and a sequence number assigned on INSERT, with the account lock held. I use the sequence and not the date because now() marks the start of the transaction and would reorder concurrent mutations.
That turns checking a wallet into a query: the balance after one entry must equal the balance before the next, and the last entry must match the account. A daily job runs this check over the last 25 hours, on the read replica, and only detects: it never fixes. Fixing a real problem goes in as an explicit adjustment recorded by an employee, because the chain is the evidence and is never edited.
Provider callbacks
Callbacks are validated with HMAC-SHA256 over the raw request body, with a constant-time comparison. Re-serializing the JSON before validating would break the signature intermittently. Each movement kind uses the right provider id as its key: the transaction id for a bet, the result id for a win and the bet id for a refund.
Jobs across replicas
Crons run in a worker role separate from the API, coordinated by a Redis lease per brand, service and job, renewed every 20 seconds and released atomically by Lua scripts. This is not exactly-once, and I don't pretend it is: coordination reduces concurrency, and the idempotency of every effect guarantees the result.
Tests
Wallet regression tests run in CI against a real PostgreSQL on every PR. The integration tests cover, among others, balance and entry landing together, a repeated external id leaving the balance unchanged, the chain closing, recovering a credit after gaming committed without crediting twice, eight concurrent duplicate bets becoming one, and about 25 withdrawal scenarios with lost replies and concurrent replays.
What is still missing
- The ledger is rolling out: it already runs in staging and reaches production in stages, with the older history brought in by an explicit backfill.
- The unique key does not protect entries without an external id, which older producers can still send.
- Serialization failures and deadlocks on the money path still surface as errors; the queue or the provider is what retries.
- The daily check only logs; it still needs to become an alert.