An operational readiness review is a scheduled, evidence-driven check that confirms whether a system, process, or asset is genuinely safe and fit to run live. It answers exactly one question: go or no-go, with every mitigation recorded against a named owner and a date. Guidance from the DOE handbook, GAO's technology readiness framework, and AWS's ORR practice all converge on the same structure.
TL;DR:
- An operational readiness review requires signed evidence like test reports, training logs, and documented rollback plans before the launch, not just assurances.
- The review should be triggered by new system launches, major changes, maintenance restarts, or regulatory demands, with scope scaled to the risk involved.
- A formal checklist should focus on observable artifacts, with scoring based on verified evidence rather than subjective confidence levels.
- Preparation involves a self-assessment at least eight weeks before the review, with a clear Implementation Plan and entry/exit criteria established upfront.
- Embedding continuous feedback and using integrated platforms like Videra can make repeated ORRs more efficient and prevent unresolved governance gaps.
Table of Contents
- What is an operational readiness review, and why does it matter?
- When should you run an ORR, and how do you scope it?
- Who needs to be in the room, and what must they bring?
- How do you plan the timeline for an ORR?
- What evidence proves you're actually ready?
- Building a checklist and scoring readiness by maturity
- Turning ORRs into a living practice, not a one-off event
- Keystone's approach to operational readiness
- Why most ORR failures are governance failures, not technical ones
- Making readiness reviews easier to run and repeat
- Sources
What is an operational readiness review, and why does it matter?
An operational readiness review is a lifecycle checkpoint, run before full operations start, that tests whether the people, processes, and systems behind a launch actually hold up under evidence rather than assurance. It is not a status meeting. Every claim in an ORR needs proof: a signed test report, a training log, a working rollback plan.
It differs from a Technology Readiness Assessment, which scores the maturity of a specific technology using Technology Readiness Levels rather than operational fitness. A TRA asks whether the technology itself is mature enough to deploy; an ORR asks whether the organisation running it is ready. Both matter, but they answer different questions at different points in a project.
An ORR typically probes four failure categories:
- System risk: does the technology perform under real load and failure conditions?
- Process risk: are runbooks, escalation paths, and rollback procedures documented and tested?
- Human risk: are operators trained, credentialed, and available on day one?
- Governance risk: is there a named authoriser who can actually say no?
AWS frames ORR as a checklist-driven process run before launch and then periodically. That periodic element gets skipped more often than it should.
When should you run an ORR, and how do you scope it?
Four triggers should put an ORR on the calendar automatically:
- New asset or system launch, where nothing has run in production before.
- A major release or change, where existing infrastructure carries new risk.
- A restart after maintenance or shutdown, where processes may have drifted.
- A regulatory or contractual requirement, where a third party demands documented proof of readiness.
Scope should scale with complexity and hazard, not with project size for its own sake. A low-risk software update needs a lighter review than a facility restart with safety implications. NRC guidance recommends running the formal ORR after acceptance testing is complete, with at least 90 days before full-scale operations to leave room for fixing what the review finds. That 90-day buffer is the detail most project schedules quietly ignore, and it is usually the reason findings get waved through unresolved.
Who needs to be in the room, and what must they bring?
A credible ORR fails without the right people bringing the right evidence, not opinions.
- Project manager — owns the overall go/no-go recommendation and closure tracking.
- Operations and maintenance leads — bring runbooks, escalation lists, and configuration baselines.
- Engineering — brings test reports and defect logs, not verbal assurance that "it's fine."
- Security — brings access control evidence and incident response readiness.
- Training leads — bring completion records, not just a training plan.
- Governance authoriser — the person with actual power to say no, present in person.
Independent reviewers, people with no stake in the launch date, add real weight here. Project teams under deadline pressure consistently under-report risk, and a fresh set of eyes tends to catch the gaps that familiarity hides. PMI's readiness-management literature makes the same case for appointing a dedicated readiness owner early rather than assembling the group only when the review date approaches.
How do you plan the timeline for an ORR?
Getting the sequence right matters more than getting the meeting itself right. A workable schedule looks like this:
- About three months before the ORR: prepare the Implementation Plan, defining criteria, team membership, and logistics.
- About eight weeks before the ORR: contractor or team performs a self-assessment using the same criteria as the formal review.
- Pre-visit walkthrough: short session to identify obvious gaps before the full team convenes.
- On-site or formal review: conduct the actual ORR based on entry criteria.
- Post-review: report findings, assign owners and closure dates, and track to resolution.
The DOE handbook's planning structure emphasizes that the Implementation Plan is critical to the schedule. Clear specification of criteria and team membership in the plan helps the on-site review proceed efficiently with fewer surprises.
Pro Tip: Run the contractor self-assessment as a genuine dry run, not a formality. Teams that treat it seriously close most findings before the formal reviewers ever walk in, which is exactly the point.
What evidence proves you're actually ready?
Entry criteria decide whether a review even starts; exit criteria decide whether the answer is go. Both need to be written down before the review, not negotiated during it.
Typical entry criteria include:
- Acceptance testing complete, with signed test reports.
- Training complete, with dated attendance or certification logs.
- Documentation available: runbooks, configuration baselines, contact lists.
- Contingency and rollback plans written and, where possible, tested.
Exit criteria are stricter. Every high-priority finding must be closed or carry an approved mitigation with a named owner and a firm date; there's no vague "will address post-launch." NRC procedure ties readiness to demonstrated test traceability rather than assurance statements, and that principle holds regardless of sector. Acceptable evidence looks like a signed readiness-to-proceed memo, a dated disaster-recovery exercise report, or a verified rollback plan someone has actually rehearsed, not just written.
Building a checklist and scoring readiness by maturity
A useful checklist splits into categories: architecture, operations and processes, security and compliance, testing, training, monitoring and alerts, and rollback or recovery. Within each category, write items as observable statements, not vague intentions. "Disaster recovery works" is not checkable. "DR runbook signed and DR exercise report dated within six months" is.
AWS's operational excellence guidance recommends scoring against objective evidence rather than subjective confidence, and tying each score to a corrective action when it falls short.
A simple maturity scale with qualitative categories is effective in practice:
- No evidence present; significant gap requiring immediate attention.
- Interim measures in place with clearly assigned ownership and planned resolution.
- Full evidence present and verified where feasible.
Anchoring each checklist item to a single named artefact rather than a subjective judgement is what actually reduces disputes and speeds up closure during a review.
Score every item this way, and the go/no-go decision stops being a debate. It becomes a count of how many items sit in "Ready" versus how many carry unresolved risk.
Turning ORRs into a living practice, not a one-off event
The biggest waste in most ORR programmes is treating each review as a standalone event, then starting from a blank checklist next time. AWS recommends explicitly feeding post-incident findings back into the ORR checklist, so every failure after launch becomes a new question asked before the next one.
Practical patterns worth adopting:
- Maintain a shared checklist repository rather than rebuilding one per project.
- Use automated configuration guardrails where possible, so some checks verify themselves.
- For immature technologies flagged in earlier assessments, attach a Technology Maturation Plan with milestones as deferred exit criteria rather than pretending the gap doesn't exist.
- Secure executive sponsorship so ORR requirements are written into architecture standards, not left optional.
Human readiness, meaning training, access, and clear decision authority, gets missed more often than technical checks, largely because it's harder to quantify than a test pass rate; consider a social engineering exposure assessment to help identify and address human-factor vulnerabilities.
Keystone's approach to operational readiness
Keystoneconsulting has spent 20 years inside healthcare, construction, and facilities management projects where governance gaps, not technical failures, cause most launch delays. The Videra platform maps workflows and evidence directly against stage gates, so readiness isn't reconstructed from memory during a review. It's already tracked. Peter and the wider Keystoneconsulting team apply this method by integrating with client teams directly rather than delivering a generic assessment and walking away, which tends to leave the readiness evidence stronger than they found it. A consultancy engagement typically slots in at the Implementation Plan stage, well before the formal review, where the criteria and evidence structure get set for everything that follows.
Why most ORR failures are governance failures, not technical ones
Most published guidance on operational readiness leans hard into checklists and technical evidence, and that's the right instinct. What it underplays is that the checklist rarely fails the review. Governance does. A finding gets logged, nobody owns it, no date attaches to it, and it survives untouched from one project to the next.
The conventional advice treats ORR as an event: book the meeting, run the checklist, get the sign-off. The DOE and NRC frameworks both point at something more useful, that readiness is a state you build over months through an Implementation Plan and a genuine self-assessment, not a verdict you extract from a single meeting.
If you take one thing from this, prioritise entry criteria over exit criteria. Teams spend enormous energy negotiating what counts as "closed enough" to pass, and far too little deciding what evidence should even be allowed into the room. Make entry criteria strict and specific, and the exit conversation mostly resolves itself, because you've already filtered out the assurance-without-evidence problem before the formal review starts.
— Peter
Making readiness reviews easier to run and repeat
There are other ways to manage ORR documentation: spreadsheets, shared drives, standalone project management tools. Most of them work fine until the checklist grows across multiple sites or sectors, at which point evidence starts living in inboxes instead of anywhere a reviewer or auditor can find it.
Keystoneconsulting built Videra specifically to fix that gap. Instead of rebuilding your checklist from scratch each time, Videra maps evidence directly to stage gates, so entry and exit criteria stay linked to the actual documents, test reports, and sign-offs behind them. Clients typically see fewer open findings at start-up, because the evidence trail is already in place rather than assembled the week before the review. Ownership of each finding is visible too, so nothing sits unresolved without a named owner and a date attached. If you're planning an ORR for a new system, a major change, or a post-maintenance restart, get in touch with Keystoneconsulting for a Videra demonstration and see how the platform fits your existing Implementation Plan.
Sources
- GAO, Technology Readiness Assessment Guide: Best Practices
- Operational Readiness Reviews (ORR) — AWS
- DOE handbook: Guide to Good Practices for Operational Readiness Reviews (ORR)
- Operational Readiness Review procedure (NRC) — Identifier P–2141