Stop getting better at firefighting

For most of my career, being good at database operations meant being good at firefighting.

Something happens in production. Queries slow down, CPU climbs, replication falls behind, storage starts growing faster than anyone planned for. Everyone gets involved. Graphs are opened, logs are searched, pg_stat_activity is inspected, someone pulls up a query plan. Eventually you find the problem, fix it, and production settles down.

If you solved it quickly, that counts as a success.

I increasingly think that is the wrong definition of success.

The best database incident is not the one you resolve in 20 minutes. It is the one that never happens.

Most problems start long before the alert fires

PostgreSQL rarely goes from healthy to broken in a second. Incident conditions usually develop quietly, over weeks or months. Statistics slowly stop representing the data distribution. Autovacuum no longer keeps up with the rate at which a table changes. An index that once had excellent selectivity becomes less useful as the data skews. A configuration that was perfectly reasonable at 200 GB is still there at 2 TB. A replication slot retains WAL for longer than expected. Privileges accumulate through years of migrations and application changes.

None of these causes an outage today.

That is exactly the problem.

Your monitoring stays green while the conditions for tomorrow's incident are already being created. Then the workload shifts a little. Traffic grows. The planner finds a cheaper execution plan. One more batch job gets scheduled.

And suddenly there is a fire.

I have seen this pattern more times than I can count. A reporting query that ran in 400 ms for two years starts taking 40 seconds on a Tuesday morning. Nothing was deployed. Nothing changed, as far as anyone could tell. What had actually happened was that a table crossed the point where the planner's join estimates no longer matched reality, and a nested loop that had been cheap for two years quietly became the most expensive thing on the server. The statistics had been drifting for weeks. Nobody was looking, because nothing was red.

Better firefighting is still firefighting

The usual response to incidents like that is more monitoring. More dashboards, more alerts, better runbooks, faster escalation.

All of those are useful, and I have built plenty of them myself. But they mostly improve what happens after a condition has become bad enough to attract attention. They shorten the time to recovery. They do not reduce the number of fires.

I would rather remove the ignition source.

If table growth and workload changes are gradually making an important query more expensive, I want to understand that before users notice the latency. If vacuum behavior is moving toward the point where dead tuples and visibility-map changes will significantly increase I/O, I want to see the trajectory instead of waiting for the I/O alarm. If the planner starts making different choices because cardinality estimates no longer match reality, the interesting question is not whether the query is slow today. It is why the economics of that execution plan changed.

That is a very different way of looking at database operations.

What we are really buying is predictability

When people talk about database tooling, they talk about features. Metrics, dashboards, query analysis, recommendations, alerts.

Those are not what I actually want to buy.

I want stability. I want to know that a database behaving well today is unlikely to surprise me tomorrow. Predictable performance, predictable infrastructure requirements, predictable cloud costs, and a security posture that does not quietly erode as roles and grants pile up over the years.

And when something is changing, I want enough warning to investigate it during normal working hours instead of discovering it because someone opened a production incident at 2:17 AM.

That sense of being in control is worth more than another dashboard.

This is also why I started building Elevarq

After more than 20 years working with PostgreSQL, I have seen plenty of incidents that looked sudden from the outside but were not. The evidence was almost always already there.

The hard part is having someone continuously watch for those small signals, understand how they relate, and recognize when today's harmless condition becomes tomorrow's problem.

A DBA can do that. But a good DBA also has migrations, upgrades, developers, architecture discussions, production requests, and another twenty databases demanding attention. They cannot continuously watch every database, every hour of every day, looking for small changes that might become important weeks later.

That is the problem we are solving with Elevarq. Not by replacing monitoring, and not by pretending every anomaly is an emergency. The goal is to continuously look for the conditions that make PostgreSQL less predictable, explain why they matter, and give the humans enough information to decide what to do about them.

I do not think the future of database operations is a better fire alarm.

It is having fewer fires.

And eventually, getting to the point where an uneventful week in production is not luck.

It is engineering.