Why does the database keep spiking?

Trigger

The CloudWatch alarm rds-cur8-cpu-utilization-high (eu-west-2) tripped four times in 24 hours — peaking at 99.2% CPU on a db.t3.large. Not a one-off; a pattern.

What the agent saw

• Performance Insights (08:40–09:00 UTC): two LiquidationsTable SELECTs at 0.81 of total DB load. Everything else was below 0.005.
• The same SQL signature dominated the earlier spikes too.
• HTTP logs showed each /admin/liquidations/v2/search taking ~32 seconds.
• Read IOPS spiked 30× while writes stayed flat and connections were calm — pure read pressure.
• The read-replica sat idle at 3–7% CPU throughout.
• 88 searches in ~40 minutes from one admin session working through 8 sellers.

Investigation

The admin liquidations search eager-loads fees, investments, withdrawals, and documents per row, and orders by requestedAt — and none of those foreign-key columns nor requestedAt are indexed. So every search runs five sequential scans on tables that grow with every payment, investment, and withdrawal. The 32-second responses then produced an impatient retry / clear / re-apply pattern that compounded into 88 requests.

Two earlier hypotheses were ruled out against the request samples: it wasn't a missing debounce (an Apply-Filters button gates refetch), and it wasn't a detail page hitting the search endpoint (every sampled request used seller filters, never an id).

The fix · #9598, #9603

• Add the five missing btree indexes. The migration is idempotent and can be applied out-of-band with CREATE INDEX CONCURRENTLY, so there's no write lock.
• Disable the admin liquidations action while a request is in flight — stops the retry storm at the source.

Verification & outcome

The load source was pinned to a single query signature with hard evidence, then indexed away. Follow-ups noted: slim the list-view payload, route admin reads to the idle replica, and retune the alarm threshold (30% → 70%) that was paging on routine admin work.

Knowledge-graph node