Roughly 41% of enterprises report data quality and legacy-integration issues as their primary blocker to AIOps working in production (AIOps Community, 2026). The number is not surprising if you have tried to deploy one: the tool lands, the alert volume either stays flat or goes up, and the vendor support response is “tune the thresholds.”
Threshold tuning is downstream of a foundation problem. The foundation problem is that most AIOps tools require clean, consistently tagged, well-correlated telemetry to do their job — and most mid-market environments do not have that before they decide to buy.
This checklist scores your environment across seven dimensions before deployment. Each dimension has a binary pass/fail question and a severity weight. Add up your unchecked items at the end.
How to use this
Score each dimension: pass (1 point) or fail (0 points). Total out of 7.
- 6–7: Proceed to deployment. Your foundation is ready; deployment work is primarily configuration and tuning.
- 4–5: Proceed with remediation plan. Identify which failed dimensions are blocking vs. non-blocking for your chosen vendor, and sequence the foundation work alongside deployment.
- 0–3: Fix the foundation first. Deploying now is likely to produce a high-noise result that undermines trust in the system. See the remediation guidance for each failed dimension below.
Dimension 1: Telemetry coverage
The check: Do at least 80% of your production services emit all three telemetry signals — metrics, logs, and traces — via a unified agent or SDK?
Why it matters: AIOps correlation requires all three. Metrics give you the anomaly signal. Traces give you the causation path. Logs give you the diagnostic detail. A tool that has two of the three produces incomplete correlation stories. You will get “something happened here” without “here is what caused it and where to look.”
How to verify:
# Datadog: check APM coverage
# In Datadog UI: APM > Service Catalog > filter env:production
# Count services with zero trace volume in last 7 days
# CloudWatch: check for services with no custom metrics
aws cloudwatch list-metrics --namespace "YourApp" | \
jq '[.Metrics[].Dimensions[] | select(.Name == "ServiceName") | .Value] | unique'
# Cross-reference against your service inventory list
Remediation if failing: Generate a service inventory from your IaC repository (every aws_ecs_service, google_cloud_run_service, or equivalent resource). For each service missing APM traces, add it to an instrumentation backlog with a target completion date. Do not deploy AIOps until the top-10 services by traffic volume are fully instrumented.
Dimension 2: Tag hygiene
The check: Is there a service: tag (or equivalent) on at least 90% of billable resources in your production account, with values from a canonical registry?
Why it matters: AIOps correlation engines — Watchdog, Davis, PagerDuty AIOps event grouping — work from the service identity embedded in telemetry tags. Untagged resources produce orphan anomaly signals that cannot be correlated with anything. Inconsistently tagged resources (five names for the same service) fragment the signal.
How to verify:
# AWS: count untagged EC2 instances
aws ec2 describe-instances \
--query 'Reservations[].Instances[?!Tags || !Tags[?Key==`service`]].[InstanceId]' \
--output text | wc -l
# GCP: count untagged Cloud Run services
gcloud run services list --platform=managed \
--format="json" | jq '[.[] | select(.metadata.labels.service == null)] | length'
A result above 10% of total resource count is a failing score on this dimension.
Remediation if failing: Build a service registry YAML (see the Watchdog post for a template). Enforce the service: tag as a required field in Terraform via lifecycle precondition blocks. Run a tagging retrospective on existing resources via AWS Config or GCP Asset Inventory to identify and remediate untagged assets.
Dimension 3: Alert signal-to-noise baseline
The check: In your last 30 days of on-call, was the ratio of actionable pages (requiring a human action) to total pages greater than 30%?
Why it matters: AIOps tools are sold as noise reducers. But they require a stable baseline of existing alert data to learn the noise pattern from the signal pattern. If your current alert regime is so noisy that less than 30% of pages are actionable, Watchdog or Davis has no clean signal to learn from. It will perpetuate the noise pattern because the noise pattern is all it can observe.
How to verify:
# PagerDuty API: pull last 30 days of incidents
import requests
headers = {'Authorization': 'Token token=YOUR_PD_TOKEN', 'Accept': 'application/vnd.pagerduty+json;version=2'}
response = requests.get(
'https://api.pagerduty.com/incidents',
headers=headers,
params={'since': '2026-05-24', 'until': '2026-06-24', 'limit': 500}
)
incidents = response.json()['incidents']
total = len(incidents)
# Count incidents that were acknowledged and had a resolution note
# vs. those auto-resolved with no action
actionable = len([i for i in incidents if i['acknowledgements']])
print(f"Actionable ratio: {actionable/total:.1%} ({actionable}/{total})")
Remediation if failing: Before deploying AIOps, run a manual signal-to-noise audit. For each alert type in PagerDuty/Opsgenie, classify it as actionable, informational, or noise. Suppress the noise class outright. This is unglamorous work but it creates the clean baseline an AIOps tool needs to be useful.
Dimension 4: Runbook automation maturity
The check: Do at least 50% of your P1 and P2 incident classes have a documented runbook with step-by-step resolution instructions?
Why it matters: AIOps tools route the right alert to the right team. If the team receives the alert with no runbook, the time-to-resolution improvement is zero. The alert arrived faster, but the diagnosis still takes the same time. Runbook maturity is the force multiplier that makes faster routing valuable.
How to verify: Pull your incident history. For the top-20 most frequent incident classes by volume, check whether each has a runbook link in PagerDuty’s service definition, Opsgenie’s team runbook, or your internal wiki. Count the gaps.
Remediation if failing: Write runbooks for the top-10 incident classes before the deployment engagement. Each runbook needs at minimum: symptoms, likely causes (ordered by frequency), diagnostic commands to confirm cause, resolution steps, and escalation path if the step-by-step fails. AIOps deployment work includes runbook stubs for the top-10 patterns as a deliverable — but the effective value compounds faster if some runbooks already exist pre-deployment.
Dimension 5: On-call ownership clarity
The check: Is there a documented map from every P1 and P2 service to a named team, with the team’s PagerDuty or Opsgenie escalation policy configured to receive alerts for that service?
Why it matters: This is the most common silent failure in AIOps deployments. The correlation layer identifies the right incident and routes it correctly by service tag. The tag maps to no team, or maps to a team whose escalation policy is misconfigured. The alert goes to a catch-all or drops. The on-call engineer never knows an incident fired.
How to verify:
# PagerDuty: list services with no escalation policy
curl -H "Authorization: Token token=YOUR_PD_TOKEN" \
-H "Accept: application/vnd.pagerduty+json;version=2" \
"https://api.pagerduty.com/services?limit=100" | \
jq '.services[] | select(.escalation_policy == null) | .name'
Cross-reference the PagerDuty service list against your service registry. Services in your registry with no PagerDuty match are the ownership gap.
Remediation if failing: The service registry (Dimension 2 fix) also resolves this. Each entry in the registry should have a pagerduty_service_id field that maps the service to an escalation policy. Make this a required field.
Dimension 6: Incident-response loop closure
The check: In the last 90 days, did at least 70% of SEV-1 incidents receive a post-incident review (PIR) with at least one action item tracked to completion?
Why it matters: AIOps tools learn from incident patterns over time. If incidents are not reviewed and action items are not closed, the same root causes recur and the AIOps tool re-learns the same failure pattern indefinitely. Loop closure is not a technology problem — it is a process problem — but it determines whether AIOps deployment produces compounding improvement or stagnant baseline performance.
How to verify: Check your incident management tool (PagerDuty, Jira, Linear) for SEV-1 incidents in the last 90 days. Count how many have a linked post-incident review document and at least one resolved action item.
Remediation if failing: Establish a PIR template and a mandatory 48-hour review meeting for every SEV-1. Assign the action items to named engineers with due dates in your issue tracker. Run two months of this discipline before deploying AIOps so that the baseline you start from is one where root causes are being addressed, not accumulating.
Dimension 7: Vendor fit
The check: Does your chosen AIOps vendor’s architecture match your primary telemetry source?
Why it matters: Each AIOps vendor is strongest on its own telemetry:
- Datadog Watchdog: best when your primary monitoring is Datadog and you want anomaly detection over metrics and APM data in Datadog.
- Dynatrace Davis: best when you want full-stack causation analysis and are willing to deploy the Dynatrace OneAgent uniformly across your environment.
- PagerDuty AIOps: best when your center of gravity is on-call workflow and you want noise reduction over events coming from multiple monitoring sources.
- Cloud-native (CloudWatch Anomaly Detector, GCP Cloud Monitoring alerting policies): best when you are a single-cloud shop with no appetite for a third-party AIOps licence.
- BigPanda, Moogsoft: best for large multi-source estates where correlation across many monitoring tools is the primary problem (typically $100k+/year spend commitment, enterprise scale).
How to verify: Draw a list of your current monitoring sources. How many are Datadog vs. not-Datadog (or Dynatrace vs. not-Dynatrace)? If 80%+ of your telemetry is in one vendor’s platform, use that vendor’s native AIOps. If your telemetry is fragmented across four systems, a correlation layer that spans them (PagerDuty AIOps, BigPanda) is likely the right fit.
Remediation if failing: The right answer is not to consolidate monitoring before deployment — that is a larger programme than AIOps deployment justifies. The right answer is to be deliberate about which vendor you start with, acknowledging that it will handle the telemetry sources it natively understands well and will require custom integration for the others.
Score interpretation
| Score | Interpretation |
|---|---|
| 7 / 7 | Proceed to deployment. The foundation is ready. |
| 5–6 / 7 | Proceed with a documented remediation plan for the failing dimensions. The deployment can start; foundation fixes happen in parallel. |
| 3–4 / 7 | Fix Dimensions 2 (tag hygiene) and 5 (ownership map) first — these two block everything else. Then reassess. |
| 0–2 / 7 | The AIOps deployment will not produce the result you expect at current foundation state. Allocate 4–6 weeks of observability-foundation work before deployment. |
Where to go next
If you want a scored readout against all seven dimensions — with access to your actual monitoring console and cloud account rather than self-assessment — the AIOps Readiness Scorecard is a 5-business-day engagement for $500. It returns a 12–15 page PDF with a specific score, the remediation priority order for your environment, and a vendor-fit recommendation. The fee is credited 100% to any AIOps Deployment engagement signed within 90 days.
If you prefer to start with the underlying observability-foundation problem, the observability foundation post makes the structural argument for why most vendor readiness assessments miss the actual blocker.