The Datadog sales engineer ran a demo of Watchdog that was genuinely impressive. Anomaly detection firing correctly on a staged environment, alert correlation across services, a clean summary of root cause. The team left the call ready to buy.
Six months later, they were turning Watchdog off.
The alert volume had gone up, not down. The on-call rotation was more tired than before the deployment. The platform lead was explaining to a VP of Engineering that the tool “wasn’t working” and they needed to look at Dynatrace instead.
The tool was working. The data it sat on was not.
The thing vendor readiness assessments do not measure
Every major AIOps vendor offers a readiness assessment. Dynatrace has a maturity model. PagerDuty has a readiness checklist. Datadog’s SEs walk you through it on a call. They are all variations of the same motion: a structured series of questions designed to identify which additional product modules you need to purchase before deployment will work.
None of them ask about your tag coverage rate.
None of them enumerate your services by whether they have a canonical service: tag value enforced by your IaC.
None of them pull your PagerDuty event history and calculate what percentage of pages in the last 30 days required a human action versus were auto-resolved with no investigation.
These assessments are sales motions. The finding is always the same: you need more of the vendor’s product.
The real readiness question is structural. It is not about which licence tier you are on. It is about whether the data pipeline that feeds the AIOps tool is clean enough for the tool to distinguish signal from noise.
What “not ready” actually looks like
The founder ran platform engineering at an India consumer commerce platform serving tens of millions of users. The observability stack was a combination of Datadog, PagerDuty, and custom CloudWatch metrics. Alert volume was high. On-call engineers had learned — correctly — that the majority of pages could be closed without investigation because the majority of pages were artifacts of deployment events, misconfigured dashboards, or metrics from resources nobody owned anymore.
The problem was not the tools. The problem was:
Roughly half the production resources had no service: tag or had a service: tag value from an old naming convention that did not match the current service inventory. A service called payments-v2 in the registry was still emitting metrics tagged service:checkout-payments. Watchdog monitored both as separate entities. Anomalies on one did not correlate with anomalies on the other.
The on-call ownership map had twelve services with no team assignment. When alerts from those services fired, they went to a catch-all escalation policy. Every engineer on rotation received them. Every engineer on rotation learned to ignore them.
Three of the top-five incident classes had no runbook. When those incidents fired, the on-call engineer opened a Slack DM to the service owner, waited for a response, and eventually resolved the incident by the same manual procedure that had resolved it the previous fourteen times.
None of these were tool problems. Replacing Datadog with Dynatrace would not have fixed tag coverage. Upgrading to PagerDuty AIOps would not have written the runbooks. Adding another licence would not have cleaned the ownership map.
Spending six weeks fixing those three things changed the on-call experience more than any tool purchase had.
The four things that actually need to exist
The observability foundation for AIOps is not complicated. It is four things, all of which require engineering discipline rather than budget:
A service registry. A canonical list of services, each with a name, an owning team, a severity tier, a Datadog or CloudWatch service: tag value, and a PagerDuty escalation policy ID. This exists in a YAML file in your infrastructure repository and is required reading for any service onboarding PR. It is not a database. It does not need a UI. It needs to be maintained.
IaC-enforced tagging. The service: tag and the team: tag are required fields on every cloud resource. Enforced by a Terraform precondition block, a CloudFormation stack policy, or a GCP organization policy — not by a plea in the README. Resources that fail tag validation do not deploy.
Deploy events flowing to your monitoring platform. Your CI/CD pipeline sends a deploy event to Datadog (or CloudWatch, or PagerDuty) every time a production deployment starts and ends. This is a five-line addition to your GitHub Actions or GitLab CI configuration. It enables Watchdog to suppress anomaly detection during deployment windows, which eliminates the single largest category of false-positive alerts in high-deployment-cadence environments.
Runbooks for the top-10 incident classes. Not comprehensive runbooks. Not runbooks for every possible incident. Runbooks for the ten incident types that fired most often in the last 90 days. Each runbook covers: symptoms, most-likely cause, diagnostic commands, resolution steps, escalation path. They live in your on-call tool (PagerDuty runbooks, Opsgenie runbook section) so they are linked from the alert, not buried in a Notion page nobody opens at 02:00.
These four things take 3–6 weeks of focused engineering time. They cost nothing in licences. They are the difference between an AIOps deployment that reduces on-call load and one that adds to it.
Why teams skip the foundation work
The honest answer is incentives.
Buying a licence is a single decision that produces a Slack announcement. The platform team delivered. The VP of Engineering can tell the board the team has AIOps now.
Cleaning tag hygiene is six weeks of unremarkable engineering work with no visible milestone. It does not produce a Slack announcement. It does not appear in a board deck. The people who most need it to happen are the on-call engineers who will benefit from it, and they are not in the room when budget decisions get made.
Vendor readiness assessments accelerate this dynamic by validating the licence decision. “Your assessment shows you’re at Level 2 maturity — you need the Enterprise tier to reach Level 3.” This is a real thing a real vendor SE will tell you. It is not wrong. It is also not the blocking issue.
The blocking issue is that the vendor’s Level 3 maturity model assumes you have clean service identity and team ownership mapping in place. The model does not say this explicitly because the vendor cannot sell you clean service identity. They can only sell you the platform that sits on top of it.
What an independent readiness assessment looks like
The 7-dimension scorecard we built for the AIOps Readiness Scorecard is different from a vendor assessment in one specific way: the first three dimensions — telemetry coverage, tag hygiene, and alert signal-to-noise — all score your existing data estate before any new tool is introduced.
A vendor assessment starts from the assumption that you are buying their tool and evaluates your readiness to use it. An independent assessment starts from the question of whether your data estate is clean enough to support any AIOps tool, and which tool is the right fit given what your estate actually looks like.
These are different questions. The first leads to a licence. The second sometimes leads to “fix the foundation before you spend the licence money.”
The honest outcome of some Scorecard engagements is: your alert volume problem is a tag-hygiene problem, not a tool problem. Spend six weeks on the foundation. Then we can talk about deployment.
That is not the answer a vendor assessment returns.
The steelman for buying the tool first
The counterargument is worth taking seriously. The tool creates the visibility that lets you prioritise the foundation work. Without a Datadog deployment, you cannot easily measure tag coverage — because measuring tag coverage requires a monitoring tool that shows you which resources are emitting metrics and which are not.
This is true. It is also a reason to have a basic monitoring deployment, not a reason to add the AIOps module before the foundation is clean. Datadog Infrastructure Monitoring tells you your tag coverage without Watchdog. CloudWatch Metrics tells you your resource inventory without adding AIOps. The foundation audit does not require the premium tier.
The trap is buying the premium tier and the foundation tooling simultaneously, then attributing poor results to the premium tier rather than to the foundation not being ready yet. The failure mode is: “we have Watchdog and it’s not working, let’s try Dynatrace Davis.” The Dynatrace Davis deployment also doesn’t work. Because the data is still bad.
The diagnostic ladder
Before your next AIOps conversation with a vendor, run this:
Pull the last 30 days of on-call pages. What percentage were acknowledged and resolved by a human action? If that number is below 40%, you have a foundation problem, not a tool problem.
Pull the list of production resources in your primary cloud account. What percentage have a service: tag with a value from a canonical registry? If that number is below 80%, your AIOps tool will learn the noise pattern, not the signal pattern.
Pull your service inventory. For how many of those services does a named team receive alerts in PagerDuty or Opsgenie? If the answer is “most but not all,” the AIOps deployment will surface the gaps by routing alerts to a catch-all that erodes trust in the whole system.
If all three of those numbers are healthy, run the 7-check readiness assessment and start the deployment conversation.
If any of them are not, you know where to start. It is not with a vendor demo.
The platform lead who turned off Watchdog eventually circled back. Not to Watchdog — to the tag coverage problem, the ownership map, the deploy event pipeline. It took six weeks. It cost nothing in new licences.
Watchdog went back on the next quarter. This time the alerts went down.