Why your database reads spike overnight when no users are online

What should be true before you start?

Start from the shape of the spike, not the size. Human traffic is uneven — it follows waking hours and product moments. A read pattern that holds a steady rhythm through the night, with nobody on the app, is a process on a schedule. Naming what runs on that schedule is the whole job.

Read cost is a measurement problem before it is an optimization problem. You cannot cut what you cannot attribute, and a database bill is a single total with no memory of which feature, screen, or background job spent it. The first job is to make the reads legible — grouped by the part of the product that caused them — so the expensive path is obvious instead of theoretical.

• Compare the read curve to your active-user curve; a mismatch points at a machine.
• List every cron job, scheduled function, and long-lived listener you run.
• Name buckets for each scheduled job so it can be told apart from user traffic.

How do you find where the reads go?

The fastest way to unmask a machine is to wrap your scheduled jobs in their own buckets. Once each job reads under its own name, an overnight spike stops being anonymous — it shows up as a single bucket climbing while user buckets sit flat.

The honest version of this measures the work already in hand. A good read meter counts the documents or rows a query already returned — it never runs an extra query, an EXPLAIN, or a profiler scan to measure, because a cost tool that itself costs reads is worse than none. Counting stays in memory, and attribution rides on the name you gave the path.

• Wrap each scheduled job, backfill, and listener in its own named bucket.
• Let the meter attribute the overnight reads to those buckets.
• Compare the read timeline to user activity; the machine bucket climbs with no users.
• Decide whether the job must read at all, or can read only what changed.
• Cut or narrow it, then confirm the overnight bucket flattens.

import { bucket } from "@cross-deck/buckets"

// now an overnight spike has a name instead of hiding in the total
export const nightlyReconcile = schedule("0 2 * * *", () =>
  bucket("jobs>nightly-reconcile", () => reconcileAll()))

Where do teams get this wrong?

The mistake is scaling the database to absorb a spike that a single disabled job would have removed.

Most read-cost surprises are not one greedy query; they are an unattributed one. A scheduled job that re-reads a collection every few minutes, a dashboard that re-scans on every refresh, or a listener that fans out on each change can outspend every user-facing feature combined — and none of it shows up until you group the reads by cause and one bar dwarfs the rest.

• Assuming a spike means growth when it means a job is re-reading data.
• Leaving a backfill or migration running long after it was needed.
• Keeping a listener attached that re-reads on every change to a busy collection.

How does Crossdeck Buckets surface this?

Crossdeck Buckets keeps reads in fine-grained time slots, so a steady overnight rhythm is visible as a pattern, and it groups each job under its own bucket — so the machine driving the spike names itself instead of hiding in the total.

The founder rule behind this is simple: a constant read pattern with nobody on the app is a machine, not a user — and machines are the easiest, cheapest reads to cut once you can see which one it is.

This is also the upgrade path, and it stays free across the step. The open-source collector shows the reads on one surface, grouped by the buckets you named — no account needed. Sign up to Crossdeck and a single SDK install adds the dimension the collector alone cannot see: which environment each read ran in — your server, your web app, your dashboard, or a mobile build — folded into the same buckets, still free. A spike stops being a guess between “is it the backend or the client?” and becomes a labelled segment you can read at a glance.

What should a healthy setup let you do?

After instrumenting, you should be able to open one view and name the top three features by read load, point to the single path driving the biggest bar, and say which environment it ran in. If that still takes a spreadsheet and a guess, the setup is not finished.

A healthy setup also makes the next change cheap to verify. Shipping an index, a cache, or a narrower query should move a specific bucket down — and you should be able to see that it did, not infer it from next month’s invoice.

• Rank features by read load and find the biggest single path.
• See which environment a read ran in — server, web, dashboard, or mobile.
• Confirm a fix moved the right bucket down, not just the bill as a whole.

What should you review after it is running?

Review the biggest bucket first — the single largest source of reads is almost always where the cheapest win lives. Then look for the rhythm that does not match your users: a steady overnight wave with nobody on the app is a machine, not a customer, and machines are the easiest reads to cut.

Treat the read meter as an operating surface, not a one-time audit. Each spike, each new feature, and each background job is a chance to confirm the cost is attributed before it compounds.

• Start at the largest bucket; that is where the cheapest win usually is.
• Watch for read patterns with no matching user activity.
• Re-check attribution whenever you add a feature or a scheduled job.

How should the whole team use it?

Read cost is not only an engineering concern. A founder watching runway wants the trend and the biggest line item. An engineer wants the exact path and environment to fix. Both are reading the same buckets, just at different depths.

When the cost is attributed by feature and labelled by environment, the conversation changes from “the database bill went up” to “this job, on the server, doubled — here is the fix.” That is the difference between a vague worry and a one-line task.

• Founder: watch the trend and the largest cost driver.
• Engineering: jump to the exact path and environment to fix.
• Everyone: reason from one attributed view instead of a single total.

Frequently asked questions