Ledger app scaling
Ledger app scaling and launch checklist
A ledger app stays fast by paginating every list, indexing the main questions, and deferring heavy work until a measured trigger fires. At launch, apply 9 rules, watch 10 signals and hold the performance targets below.
Last checked
Rules to apply at launch
- Idempotency: Posting uses one transaction: insert idempotency row, lock affected accounts with SELECT ... FOR UPDATE in ascending account_id order (or run at SERIALIZABLE with bounded retry on 40001), insert entry + lines, update balance_minor and velocity counters, commit. No read-modify-write across transactions, no balance checks done outside the lock.
- Idempotency: Every mutating money endpoint requires an Idempotency-Key; (initiator_id, idempotency_key) is unique on journal_entries and transfers. Replays return the original result, a conflicting body returns 422. Outbound processor calls pass their own derived idempotency keys so a retried saga step cannot double-charge or double-pay.
- Queues: System accounts (processor clearing, fee income) are hot rows; their balance is NOT updated per entry. Postings to them write journal lines only, and their balances are derived by a periodic rollup job, so user postings never contend on one row.
- Pagination: Activity and statements use keyset pagination on (created_at, id) over journal_lines(account_id, created_at DESC, id DESC); page size <= 100; never OFFSET and never a COUNT(*) over the ledger.
- Caching: Balance reads come from ledger_accounts.balance_minor (O(1)), never SUM over journal_lines at request time; available = balance - active holds from the partial index. Only the posting function writes the cache, and an hourly job proves cache = sum(lines).
- Rate limits: Per-user and per-IP limits on transfer creation, deposit creation, withdrawal and login; per-user velocity limits (count, amount, new counterparties) enforced inside the posting transaction and failing closed; per-tenant global withdrawal cap per day.
- Fanout: Webhooks are verified, stored (unique provider+event_id) and acknowledged in < 1 s; processing is asynchronous and idempotent; out-of-order events are handled by reading the current object from the processor, not by trusting arrival order.
- Data growth: journal_lines and journal_entries are range-partitioned by month once above ~20M rows; old partitions are detached to read-only archive but never dropped. Statements and reconciliation scan date ranges so they touch only the relevant partitions.
- Storage: Statements and exports are generated off-peak by batch jobs into a private bucket with signed short-lived URLs; per-user generation is capped, and old PDFs are regenerable so retention of the bucket is bounded.
Performance targets
| What to measure | Target |
|---|---|
| Transfer API (internal, between two wallets) | p95 < 150 ms end-to-end, p99 < 400 ms; one transaction, <= 8 statements |
| Balance read | p95 < 30 ms; single indexed row read, no aggregation |
| Account activity page (50 lines) | p95 < 80 ms via keyset on journal_lines(account_id, created_at DESC, id DESC) |
| Webhook acknowledgement | p99 < 500 ms; processing asynchronous |
| Saga completion for processor-backed transfer | p95 < 60 s to settled or failed state with compensation |
| Posting throughput per tenant database | >= 200 entries/s sustained with retries on serialization failure < 1% |
| Reconciliation run | daily run completes < 15 min for 1M entries; breaks surfaced the same day |
What to watch
- Metric entries posted total and post latency ms: by kind and outcome; latency histogram includes lock wait time.
- Metric serialization retries: count of 40001 retries and lock-timeout failures; rising trend means hot accounts.
- Metric ledger imbalance minor: must be exactly 0 for the sum of all debits minus credits, per currency, every hour.
- Metric reconciliation break count and amount: open breaks and net difference by provider and age.
- Metric limit rejections: rejections by limit type; spikes indicate fraud or a misconfigured tier.
- Metric saga stuck age: age of the oldest transfer not in a terminal state.
- Log money movement audit logs: request id, idempotency key, entry id, account ids hashed, amount and currency; never card data, tokens or document content.
- Trace request -> post entry -> processor -> webhook trace: one trace id from API call through saga steps to settlement.
Recurring work
- run_transfer_saga: runs queue (jobs) enqueued on transfer creation and every 1m for stuck transfers. Guard: resume from transfers.step, each processor call carries its own idempotency key, attempt cap 5 with exponential backoff, on terminal failure post compensating entry and release hold.
- process_processor_event: runs queue (jobs) after webhook insert. Guard: unique (provider, event_id), post entry with the event id as idempotency key, dead-letter after 5 attempts with an alert.
- expire_holds: runs cron 1m. Guard: bounded batch with SKIP LOCKED, release in the same transaction that marks expired.
- reconcile_processor: runs cron daily 02:00 UTC. Guard: one run per (provider, period_date); read-only against the processor; any break raises an alert and blocks new payouts for that provider.
- verify_ledger_integrity: runs cron hourly. Guard: recompute balances from journal_lines in bounded account batches; assert every entry balances; on mismatch freeze affected accounts and page.
- generate_statements: runs cron monthly + queue on demand. Guard: idempotent per (account_id, period_start); closing balance must tie out before storing.
Add when the numbers say so
- Read replicas: Statement generation, activity history and reporting move to a read replica once primary CPU > 60% sustained or history read p95 > 200 ms; posting, balance checks and the saga always use the primary.
- Data growth: Shard by account owner across tenant databases when a single database exceeds the write ceiling; cross-shard transfers are modelled as two single-shard entries through an in-transit account (saga), never a distributed transaction.
- Realtime: Push balance and transfer updates to the client by SSE backed by Postgres NOTIFY in phase 1 of the UI; a dedicated websocket tier and an outbox publisher to a message broker are phase 3 additions when downstream consumers appear.
Deferred until it is needed
| Deferred | Adopt it when |
|---|---|
| Sanctions/PEP screening and transaction-monitoring service | monthly volume > partner threshold or regulated launch market. |
| Read replicas for history and statements | primary CPU > 60% sustained or read p95 > 200 ms. |
| Message broker and outbox publisher (Kafka or similar) | > 3 downstream consumers or > 2k events/s. |
| HSM / managed KMS for field encryption keys | regulated customer or SOC 2 Type II audit. |
| Database sharding by account owner | >= 500 posts/s sustained or storage > 70% of volume ceiling. |
On Forge64
Forge64 matches your description to one of 12 reference architectures, writes the plan and data model into the project, generates the app, and runs a quality gate over the code before you see it. Each app gets its own Postgres database and sign-in with no per-user fee. Hosting on Forge Cloud and Forge Payments are rolling out.
Every metered resource on Forge64 has a hard cap, so a traffic spike stops at your limit instead of becoming a surprise bill. This is a reference design for the category, not a promise that every item is generated today. Built-ins such as email, background jobs, file uploads and search are coming soon, so anything here that needs them is something you add yourself for now.
FAQ
- When should I worry about scale for a ledger app?
- Not before a measured trigger fires. For example, for "Sanctions/PEP screening and transaction-monitoring service": monthly volume > partner threshold or regulated launch market.
- What performance targets should a ledger app aim for?
- Transfer API (internal, between two wallets): p95 < 150 ms end-to-end, p99 < 400 ms; one transaction, <= 8 statements. Balance read: p95 < 30 ms; single indexed row read, no aggregation. Account activity page (50 lines): p95 < 80 ms via keyset on journal_lines(account_id, created_at DESC, id DESC).
- Does Forge64 cap my usage costs?
- Yes. Every metered resource has a hard cap, and paid plans add an overage spend cap that you set.
Build your app with Forge64
One prompt to a running app with auth, database, hosting and payments. Start free, no credit card required.
Start building for free