Most enterprise regression suites were sized during an implementation project, back when the program controlled its own dates. A team wrote a pack that covered every process going live, agreed on a window long enough to run all of it, and signed off. Years later, that pack is often still the pack. What moved underneath it was the calendar. The platforms it was written against now publish their own release dates, on their own schedule, and nobody went back to ask whether the window still fits.
The Cadence the Suite Was Built For Is Gone
None of this is hidden. Salesforce publishes three seasonal releases a year, Spring, Summer and Winter, with the dates posted well in advance. ServiceNow ships two family releases a year on a timeline it announces ahead of time. SAP delivers two feature releases a year for S/4HANA Cloud public edition, in February and August, with maintenance updates in between.
Now put a full regression cycle next to that. If the cycle takes weeks of calendar time and the release date is fixed by someone outside the program, the arithmetic stops working. Nothing has broken yet. The suite still passes when it runs. It just can’t finish before the next thing lands.
The Skip Nobody Writes Down
What happens next is predictable and almost never documented. The suite gets partially run. Somebody picks the subset, usually the person with the most context and the least time to spare. The tests that get skipped are skipped for a reason that lives in one person’s head, and the decision leaves no trace anyone can review later.
That is the real cost, and it is not measured in hours. Coverage stops being a property of the suite and becomes a property of whoever built the run list that week.
What Risk Actually Means in a Test Suite
Risk here is not a feeling about which tests matter. It is the combination of two things that can be estimated separately, by different people, and written down.
Two Inputs, Deliberately Kept Apart
The first is impact: what it costs the business if this process fails in production. An order that cannot be created. An invoice that posts to the wrong account. A payroll run that stops. The second is change exposure: how often this area of the system actually changes, whether that change comes from a platform release, a configuration decision, or the team’s own backlog.
Kept apart, both are answerable. A finance lead can tell you what a failed invoice run costs without knowing anything about test automation. A technical lead can tell you which objects were touched in the last few releases without ranking them by business value. Merged into one vague priority column, both answers get worse.
Building the Ranking
The ranking is a small artifact. In most enterprise landscapes it is one sheet, a few dozen rows, and four columns. What makes it work is not sophistication, it is that the same four columns get filled in for every row, including the rows nobody feels strongly about.
Start From the Business Process
The unit of ranking is the business process, not the test case. Order to cash, procure to pay, hire to retire, incident to resolution. Test cases hang off those processes, and in a mature landscape there are far more test cases than anyone will ever rank by hand. Processes are countable, and the business already has opinions about them, which means the scoring conversation can actually happen.
Score Impact and Change Exposure Separately
Give each process a score for impact and a second score for change exposure. Keep the scale short. A three point scale forces a decision. A ten point scale invites negotiation. Then write the reason next to the score, in one line. The reason is what survives the argument six months later, when the person who set the score has moved to another program.
Let the Score Assign the Tier
This is the step teams skip, and it is the step that makes the ranking hold. Once the two scores exist, the tier assignment is mechanical. High impact plus high change exposure lands in the top tier. High impact plus low change exposure lands one tier down. The run list stops being something to relitigate before every release, because the argument moved up to the scoring criteria, where it belongs and where it only has to happen once.
What the Tiers Do on Release Day
Three tiers is usually enough. More than that and the boundaries blur, and a tier whose boundary is unclear ends up being run “just in case”, which is how the suite grew in the first place.
Tier 1: Every Release, No Exceptions
The top tier runs on every release, and it runs whether or not anyone believes the release touches it. That is the whole point of it, so this tier has to stay small enough that no exceptions is a credible promise. If it takes longer than the gap between the release landing in a preview environment and the day it reaches production, it is not a tier. It is the old suite wearing a new label.
Tier 2: Only When the Change Touches It
The middle tier is conditional. It runs when the change actually reaches its area, which means somebody has to be able to say what the change reached. That is a discipline question more than a tooling question, and it is exactly where the ranking connects back to the release notes.
Tier 3: On a Calendar, Not on a Release
The bottom tier still gets tested. It just stops riding the release cadence. Once a quarter, twice a year, whatever the score justifies. Saying that out loud is what separates a ranked suite from a suite that is quietly being skipped. Everything still runs. The frequency is now a decision on paper instead of a side effect of the clock.
Feeding the Ranking From the Release Notes
A ranking that only gets revisited once a year is a document. A ranking that gets revisited every time a vendor publishes its release notes is a working control, and the input is already sitting in the inbox.
Reading a Release Note as a Test Input
Every one of these platforms publishes what changed before the change arrives. Most QA functions read those notes as documentation. Read as a test input, a release note answers three questions:
- Which modules or objects were touched
- Which changes are on by default and which have to be enabled
- Which behaviors are flagged as retiring
Map those answers against the process list and Tier 2 selects itself. The question moves from “what should we run this time” to “what did the note say changed”, and the second one has an answer.
The Test Nobody Wants to Retire
Every mature suite carries tests that exist because something went wrong once, years ago, on a system that has since been replaced. They pass every time. Nobody will delete them, because deleting a test feels like accepting risk while keeping one feels free. It is not free. It is a share of every cycle, forever.
The ranking gives that conversation a structure. The test does not get deleted because someone decided it was useless. It moves to Tier 3, because its process scored low on both inputs. That is a decision anyone can read back, and it is reversible in one line.
What the Program Looks Like Once It Holds
None of this shows up as a dramatic before and after. It shows up as a set of small changes in how release weeks feel, and two of them are worth naming.
The Suite Gets Smaller and the Coverage Gets Explainable
The first visible result is that the cycle fits inside the window again. The second one matters more. When someone asks why a given area was not tested for this release, there is an answer with a name and a date attached, instead of a shrug. Coverage becomes something you can defend, which is a different thing from coverage being high.
Automation Lands Where It Pays
A ranked suite also makes the automation investment obvious. Tier 1 is the shortlist: small, stable, run on every single release, and therefore the place where an automated test pays back fastest. Teams that automate before ranking tend to automate whatever was easiest to automate, which is rarely what runs most often.
Conclusion
The release calendar is not going back to what it was. The platforms will keep publishing dates, and those dates will keep landing whether or not a full regression cycle fits between them. What a QA function still controls is what it chooses to run, and whether that choice is written down.
A risk ranking is not more testing. It is the same testing, ordered so the cycle fits the calendar the business is actually on, and so the parts that run less often sit on a list somebody signed rather than in a subset somebody improvised on a Thursday afternoon.
Rethinking regression scope for a release calendar you do not control?
Our Testing Services team works through this with enterprise QA functions running on SAP, Salesforce and ServiceNow. Tell us how long a full cycle takes you today, and we will walk through what a ranked version of it would look like.
