Stablecoin payouts that send twice, and how to stop them

The payout bug I find most often isn’t subtle. A worker sends a stablecoin transfer, the RPC call times out, the job framework retries, and the retry signs and sends a second transfer. Both confirm. The recipient now has twice what they were owed, and unless they choose to send it back, the money is gone.

Nobody writes this bug on purpose. It comes from a reasonable instinct carried over from web development: if a request fails, retry it. On-chain, “failed” is often a lie. The call timed out, but the transaction was already in a mempool and went on to be mined. This guide is about building payouts that are safe to retry: a durable intent, provider idempotency keys, signing exactly once, and the chain-specific rules for EVM nonces and Solana blockhashes.

No. 01The ambiguous timeout is the normal case

Every money-moving call has three outcomes, not two: it succeeded, it failed, or you don’t know. The third is the dangerous one, and it is common:

• The RPC node accepted the transaction and propagated it, then the HTTP response was lost.
• The provider API created the transfer, then your load balancer cut the connection at 30 seconds.
• The worker process was killed between broadcasting and writing the result to the database.
• The job queue redelivered a message because the acknowledgement was late, and a second worker picked it up.

In each case, a naive retry creates a new payment. The rule that follows: an unknown outcome must be resolved, never retried blindly. Resolving means asking the chain or the provider what actually happened to this payment, which is only possible if you recorded enough to identify it before you sent it.

No. 02Record the intent before anything moves

A payout starts life as a row in your database, written before any key is touched or any API is called. That row is the single source of truth for whether the payment should exist, and it carries a unique idempotency key derived from the business event, not generated at send time:

CREATE TABLE payouts (
  id               uuid PRIMARY KEY,
  idempotency_key  text NOT NULL UNIQUE,   -- e.g. "withdrawal:8f2c...": one per business event
  chain            text NOT NULL,
  token            text NOT NULL,
  destination      text NOT NULL,
  amount           numeric(78,0) NOT NULL, -- base units, never floats
  status           text NOT NULL DEFAULT 'created',
  nonce            bigint,                 -- EVM: assigned once, never changed
  raw_tx           bytea,                  -- the signed bytes, stored before broadcast
  tx_hash          text,
  provider_ref     text,                   -- provider transfer ID, if using an API
  updated_at       timestamptz NOT NULL DEFAULT now()
);

The unique constraint on idempotency_key is what stops a double-submitted withdrawal form, a replayed webhook or a retried API call from creating a second payout. If the key is derived from the withdrawal request ID, the second insert fails and the caller gets the existing payout back.

The status moves through a small state machine, and only forwards:

• created: the intent exists. Nothing has been signed.
• signed: the raw transaction and its hash are stored. It may or may not have reached the network.
• broadcast: at least one node accepted it.
• confirmed: it is in a block at your finality threshold (see the finality guide).
• failed: you have proved it can never land. Only this state allows a new attempt.

The important property is that signed and broadcast are treated identically by recovery code. Once signed bytes exist, the payment might be on its way, and the only safe actions are to re-send those same bytes or to check whether they landed.

No. 03Exactly one worker owns each payout

Queues deliver at least once, so two workers will eventually hold the same job. Let the database decide who owns it. In PostgreSQL, FOR UPDATE SKIP LOCKED lets many workers pull from the same table without blocking each other or claiming the same row:

BEGIN;
SELECT id FROM payouts
 WHERE status = 'created'
 ORDER BY id
 LIMIT 10
 FOR UPDATE SKIP LOCKED;
-- ...sign each claimed payout and store raw_tx, tx_hash, nonce...
UPDATE payouts SET status = 'signed', raw_tx = $1, tx_hash = $2, nonce = $3, updated_at = now()
 WHERE id = $4 AND status = 'created';   -- conditional: a second writer changes nothing
COMMIT;

Keep the signing step inside that transaction and the network call outside it. Signing is local (or a quick call to your signer service, as described in the key management guide), so it can sit inside the lock. Broadcasting is slow and uncertain, so it happens after commit, from the stored bytes. If the worker crashes after commit, a recovery job finds the row in signed and re-sends exactly what was stored.

This is the outbox pattern applied to payments: the decision to pay and the bytes that implement it are committed together, and a separate step delivers them to the outside world as many times as needed.

No. 04EVM: the nonce is your idempotency key

On Ethereum and other EVM chains, each account’s transactions carry a sequential nonce, and the network will mine at most one transaction per nonce. That is a built-in idempotency mechanism, and the classic double-payout bug is throwing it away by letting the library pick a fresh nonce on each retry.

• Assign the nonce yourself, once. Keep a nonce counter per hot wallet in your database, allocate from it inside the signing transaction, and store the nonce on the payout row. Don’t call getTransactionCount(address, "pending") on every send: it depends on what one node’s mempool currently holds and returns the same value to two concurrent workers.
• Re-broadcast, don’t re-sign. On timeout, send the stored raw_tx again. Nodes treat it as a duplicate if they already have it. Then look up tx_hash.
• Replace with the same nonce. If a transaction is stuck because fees rose, sign a replacement with the same nonce and a higher fee. Most clients require a meaningful bump (Geth’s default is 10%) before accepting a replacement. Because both share a nonce, at most one can be mined. Record every replacement’s hash on the payout, because the one that lands may not be the latest.
• Cancel with the same nonce. To abandon a payout, send a zero-value transaction to yourself with that nonce. Only once that cancel is confirmed is the original provably dead and the payout failed.

// ethers v6. Resolve a payout whose broadcast outcome is unknown.
async function resolveEvmPayout(provider, payout) {
  for (const hash of payout.txHashes) {          // original plus any replacements
    const receipt = await provider.getTransactionReceipt(hash);
    if (receipt) return receipt.status === 1 ? { state: "landed", hash } : { state: "reverted", hash };
  }
  const mined = await provider.getTransactionCount(payout.from, "latest");
  if (mined > payout.nonce) {
    // The nonce was used by a transaction we don't recognise. Stop and page someone.
    return { state: "nonce_consumed_by_unknown_tx" };
  }
  await provider.broadcastTransaction(payout.rawTx).catch(() => {}); // same bytes, safe to resend
  return { state: "pending" };
}

A reverted transaction also used its nonce. Treat it as failed only after investigating why it reverted. A new attempt then gets a new nonce and a new intent version, never a silent retry.

No. 05Solana: wait for the blockhash to expire

Solana has no account nonce. Instead, every transaction includes a recent blockhash, and the network rejects a transaction whose blockhash is too old: roughly 150 blocks, which is about a minute or two. That gives you a different but equally strict rule.

• When you sign, store the transaction signature and the lastValidBlockHeight returned by getLatestBlockhash.
• Until the current block height passes lastValidBlockHeight, re-send the same signed transaction and poll getSignatureStatuses.
• Only after the block height has passed that value, and the signature still shows no status (query with searchTransactionHistory: true if needed), is the transaction provably unable to land. Then, and only then, sign a new one.

const status = await connection.getSignatureStatus(payout.signature, { searchTransactionHistory: true });
if (status.value?.err === null) return "landed";        // check commitment against your finality policy
if (status.value?.err) return "failed_on_chain";
const height = await connection.getBlockHeight("confirmed");
if (height <= payout.lastValidBlockHeight) return "resend_same_bytes";
return "expired_safe_to_resign";                         // the old transaction can never land

The common bug is the opposite order: the send times out, the code fetches a fresh blockhash and signs again while the first transaction is still valid. Both land.

If you need to sign a payout and broadcast it later, for example after an offline or multi-party approval, use a durable nonce account. The transaction then references the nonce account’s stored value instead of a recent blockhash and stays valid until that nonce is advanced. The idempotency rule becomes: advancing the nonce (which any transaction using it does as its first instruction) invalidates every other transaction signed against the same value, so only one can ever execute.

No. 06Provider APIs: pass the key, and keep it

If a provider moves the money for you, its idempotency mechanism does the job of the nonce. Each one works a little differently:

• Stripe takes an Idempotency-Key header on POST requests. It saves the first response for a key, including errors, and replays it for retries. It compares parameters and errors if a reused key comes with a different request. Keys can be pruned once they are at least 24 hours old.
• Circle developer-controlled wallets take an idempotencyKey field in the JSON body of transfer requests, and it must be a UUID v4. It is required, not optional.
• Coinbase CDP (v2 API) takes an optional X-Idempotency-Key header on POST requests, which must be a UUID v4. A key is unique within a rolling 24-hour window, and reusing it with different parameters returns an idempotency_error.
• Bridge requires an Idempotency-Key header on POST requests, including transfer creation. A retry with the same key within 24 hours returns the original response; reusing a key after that window returns a 422.

Three mistakes recur across all of them:

• Generating the key at send time. idempotencyKey: uuid() inline in the API call gives every retry a new key, which disables the protection. Generate it when the payout intent is created and store it on the row.
• Retrying past the window. Every provider above limits how long a key is remembered, typically about 24 hours. A retry queue that backs off for two days can create a duplicate with the correct key. Before retrying an old payout, look it up by your reference or the stored provider_ref.
• Losing the provider reference. Write the provider’s transfer ID to the payout row as soon as you get it, in the same step that records the response. Without it, reconciliation has to match on amount and destination, which is ambiguous for repeat payments.

Format matters too: Circle and Coinbase reject keys that are not UUID v4. If your business key is a string like withdrawal:8f2c, derive a UUID from it deterministically (a UUID v5 of the business key is one option) or store a generated UUID v4 alongside it.

No. 07Assume one will slip through, and catch it

Even a careful design has edges: a manual payout sent from a wallet UI, an old worker version still running during a deploy, an operator re-running a failed batch. Detection is the second line of defence:

• Reconcile outgoing transfers to intents. Every outgoing transfer from a hot wallet, read from the chain, must match exactly one payout row by hash. An on-chain transfer with no row, or two transfers matching one intent, opens an incident.
• Alert on near-duplicates. Two payouts to the same destination for the same amount within a short window aren’t always wrong, but they deserve a look before the next batch runs.
• Cap the blast radius. Per-destination and per-hour limits in the signer’s policy layer turn a runaway retry loop into a few duplicates instead of an emptied wallet.

The ledger drift section of the main guide covers the wider reconciliation picture. For payouts specifically, the check is simple: count on-chain transfers per intent, and the answer must always be zero or one.

Test planTen tests for your payout pipeline

• Submitting the same withdrawal request twice creates one payout row and one transfer.
• Killing the worker right after broadcast, then running recovery, results in exactly one transfer on the explorer.
• Making the RPC call throw after the node accepted the transaction results in exactly one transfer.
• Two workers started on the same queue message produce one signed transaction.
• Two concurrent EVM payouts from the same wallet get different nonces, and a retry of either reuses its stored nonce.
• A stuck EVM payout replaced with a higher fee lands once, and the payout row records whichever hash was mined.
• A Solana payout whose send times out is not re-signed until the block height passes its lastValidBlockHeight.
• A provider API call that times out is retried with the same stored idempotency key, and the provider shows one transfer.
• A payout retried after the provider’s 24-hour key window is resolved by lookup, not by a new create call.
• An outgoing on-chain transfer with no matching payout row raises an alert within minutes.