Three years into the Consumer Duty, most firms have built an outcomes testing capability of some kind. Files are being reviewed, failures are being logged, and MI is reaching committees. Yet in our work across the sector, one question consistently separates mature programmes from the rest: when testing finds a poor outcome, what happens next?
Identifying an issue is only half the job. The Duty requires firms to act where they identify foreseeable harm, and to rectify it, including through redress where appropriate. The value of outcomes testing is therefore measured not by the number of poor outcomes recorded, but by the quality of the actions it drives and the evidence that those actions worked.
In practice, many firms struggle to articulate what an appropriate action really looks like, and during our recent Outcomes Monitoring Webinar only 14% of attendees said they can trace outcomes MI through to root cause, action and verified improvement.
The challenge: one case, or a symptom?
Every poor outcome identified presents a fork in the road. At one end sits the individual action: put it right for this customer , and give feedback to the colleague(s) who handled the case. At the other end, sits the systemic action: identify and remediate every customer potentially affected in the same way, and uplift the policy, procedure, systems or training that allowed the failure to happen.
Both are legitimate responses. The difficulty is knowing which is proportionate, and we see firms fall into three recurring traps:
- Defaulting to the individual. “Feedback given to agent” becomes the standard closure note. It is quick, easy to evidence and feels like action, but if the root cause sits in a process gap, an unclear script or a system limitation, the same failure will recur with the next colleague and the next customer.
- Escalating everything. At the other extreme, a single fail triggers a request to rewrite a procedure or retrain a whole department. Change fatigue sets in, genuinely systemic issues compete for attention with one-off errors, and action logs grow faster than they can be closed.
- Describing activity, not outcome. Actions such as “review the procedure” or “raise awareness” are common. They are difficult to argue with, but equally difficult to complete, evidence or test. Without a clear end state, there is no way to know whether the customer outcome has actually improved.
The root of all three is the same: the action is chosen before the issue is properly understood. A good action is the end point of a structured line of reasoning, not the starting point.
Step 1: Pin down the issue and the outcome risk
A poor outcome is not, in itself, an issue statement. “Case failed on vulnerability” reveals very little about why a poor outcome has occurred. Good practice is to articulate the issue precisely enough that someone outside the testing team can understand what went wrong, why it matters and what they need to do about it.
That means separating three things that are often blurred together:
- What happened: the specific failure observed on the file, for example a customer disclosed a change in health during a collections call and the disclosure was not recorded or acted on.
- Why it happened: the likely root cause. Was this a one-off lapse by an otherwise competent colleague, or does the procedure fail to tell colleagues what to do? Does the system make it difficult to record a disclosure? Is the training silent on this scenario?
- The customer outcome risk: what this means for the customer’s actual outcome. In this example, the customer may not receive appropriate forbearance, may be contacted in a way that does not reflect their circumstances, and may suffer actual, foreseeable harm as a result.
Root cause is the hinge on which the whole action turns. A failure rooted in individual judgement points towards individual feedback. A failure rooted in process, system or training points towards something broader, regardless of how many times this has been seen so far.
Step 2: Quantify the impact for the customer and the population
Once the issue is understood, the next question is scale. This needs answering at two levels.
At individual level, what did this failure mean for this customer? Did they pay more than they should have, miss out on support they were entitled to, or experience delay, stress or repeated contact? Where a financial loss can be calculated, it should be. Where it cannot, the nature and duration of the impact should still be considered to determine whether the non-financial impact on the customer has been material enough to warrant further action to be taken (e.g. the payment of compensatory distress and inconvenience redress).
At population level, how many other customers could have been affected in the same way? This is where outcomes testing earns its keep. A well designed (ideally statistically significant) sample allows firms to extrapolate a failure rate to the wider population with a stated level of confidence. Even where a sample is not statistically significant, firms can usually identify the population at risk: every customer who went through the same journey, under the same procedure or system, during the period in question.
The answer to this question often decides the action. One poor outcome in a sample of 100 with an identified individual root cause may warrant feedback and a single-case remediation. The same poor outcome, traced to a procedural gap that applied to 20,000 customers over 18 months, is a remediation exercise, a procedure change and likely to require escalation via the relevant governance forum.
Crucially, firms should resist waiting for a pattern to emerge before asking the population question. If the root cause is systemic, a single poor outcome may be the only one the sample happened to catch.
Step 3: Categorise the harm
Not all poor outcomes are equal, and a consistent harm taxonomy is what allows firms to prioritise actions and compare issues across journeys. Two distinctions do most of the work.
Potential versus actual harm. Has the customer already suffered a detriment, or has the firm been exposed to a risk that has not yet crystallised? A missed affordability check on a loan that is being repaid without difficulty is potential harm; the same failure on a loan now in arrears is actual harm. Potential harm still demands action, typically to prevent recurrence, but actual harm usually demands that it’s put right for the customer.t.
Financial versus non-financial harm. Financial harm, such as overcharging, incorrect interest or fees, or a missed refund, can usually be calculated and redressed directly. Non-financial harm, such as distress and inconvenience caused by delays, repeated contact or a failure to recognise vulnerability, is harder to quantify but no less real. Firms should be prepared to recognise it, including through a distress and inconvenience payment where appropriate.
Actual
Financial: Customer charged a fee they should not have paid. Typical response: refund with interest, then population review.
Distress and inconvenience: Customer in vulnerable circumstances chased repeatedly after disclosure. Typical response: apology and D&I payment, then review of the contact strategy.
Potential
Financial: Incorrect rate applied but not yet charged. Typical response: correct before it crystallises, then fix the control.
Distress and inconvenience: Disclosure not recorded, but no further contact yet. Typical response: update the record, contact the customer, address the root cause.
A single issue can sit in more than one quadrant, and the categorisation should be revisited as the population analysis develops.
Step 4: Define a proportionate action
With the issue, root cause, scale and harm understood, the right action usually becomes far clearer. In most cases it will have more than one layer:
- Put it right for the customer. Where testing identifies actual harm on a file, the individual customer should be remediated, regardless of what else happens.
- Address the cause at the right level. Colleague feedback where the cause is individual; procedure, system, script or training change where it is not. Often both are needed.
- Find and fix the wider population. Where the root cause is systemic, identify every customer exposed to it and remediate those who suffered harm.
Whatever the layers, a good action shares a few common characteristics. It is specific, describing what will change rather than what will be reviewed. It has a named owner with the authority to deliver it. It has a realistic deadline proportionate to the harm. And it has a clear definition of done, including the evidence that will demonstrate completion.
Compare “remind team of vulnerability process” with “update the collections procedure to require a vulnerability flag and a pause in contact within 24 hours of a disclosure; deliver targeted training to all collections colleagues by 31 January; owner: Head of Collections”. The second can be completed, evidenced and, most importantly, tested.
What about good outcomes with areas for improvement?
It can be tempting to treat a case that achieves a good outcome as requiring no further action, even where we have identified weaknesses in the journey along the way. In our view, those observations still warrant an action. A failure that did not cause harm on this occasion is, in effect, a near miss: the same gap in a procedure, script or system could produce actual harm for the next customer whose circumstances are slightly different.
It is also worth asking why the outcome was good. Often the answer is that a business-as-usual control caught the problem part way through the journey, for example a customer complained and was remediated, or a quality check (QC) on one of the calls identified the error and it was corrected. That is a positive reflection of the firm’s controls, but it may not fix the underlying cause, and customers who do not complain, or whose calls are not selected for QC, will not benefit from the same safety net. Recording these findings, assessing their root cause and agreeing a proportionate action ensures that outcomes testing addresses the journey itself, rather than relying on downstream controls to catch failures after they have occurred.
Closing the loop: completed is not the same as effective
An action log full of green ticks is reassuring, but it answers only half the question. The Consumer Duty is concerned with outcomes, and a closed action is only valuable if customer outcomes have actually improved as a result.
Good practice is to build a structured follow-up into the lifecycle of every material action, typically through follow-up reviews that ask two distinct questions:
- Has the action been completed? Is there evidence that the procedure was updated and published, the training delivered and attended, the system change deployed, and the affected customers identified and remediated? This should be verified against evidence, not taken on the word of the action owner.
- Has it had the intended effect? Once the change has had time to bed in, a targeted sample of cases from the same journey should be re-tested against the same criteria. If the issue is no longer evident, the action has likely worked. If it has not, the firm has learned something important: the root cause was misdiagnosed, the action was not proportionate, or the change has not landed in practice.
The second question is the one most often skipped, and it is where the greatest value lies. A retrained team that continues to miss vulnerability disclosures tells a firm far more than the training attendance record ever could. Equally, a successful follow-up review provides exactly the kind of evidence Boards and the FCA want to see: not simply that the firm acted, but that its actions improved outcomes.
Follow-up reviews should be scheduled at the point the action is agreed, with timing that reflects the harm involved, and the findings should feed back into the testing programme. Issues that persist become higher-priority test areas; issues that are resolved can be monitored at a lower intensity.
From findings to better outcomes
Outcomes testing is only as valuable as the change it drives. Firms that get this right treat every material observation as the start of a disciplined process: understand the issue and its root cause, size it at individual and population level, categorise the harm, define a specific and proportionate action, and then go back and test whether it worked.
That discipline is what turns a testing programme from a record of what went wrong into genuine evidence that the firm is identifying and fixing poor outcomes, which is ultimately what the Consumer Duty asks of it.
How Assure 4 supports the right action, case by case and at scale
The discipline described above is easy to set out but hard to apply consistently, particularly when reviewers are working through high volumes of files under time pressure. Assure 4, our AI-enabled outcomes testing solution, is designed to build that discipline into every assessment.
At case level: from finding to action
Each Assure 4 assessment does more than record a pass or fail. For every finding, it sets out the concern, the evidence behind it, the impact on the customer, a categorisation of the harm and a recommended mitigation. Each finding is also attributed to a driver, distinguishing failures rooted in individual colleague behaviour from those rooted in policy, process or controls.
The below is an example of an output from an example overdraft arrears journey assessed by Assure 4.
Poor – Policy – Inadequate Early Signposting to Debt Advice
Concern: The firms early communications (SMS and email) when the customer neared his overdraft limit did not include signposting to free and impartial debt advice services, contray to best practice and the intent of the Collections Procedure. The policy did not require such signposting at the earliest stage, resulting in a missed opportunity for early intervention.
Impact: The customer was not made aware of external support options until his situation had deteriorated into an unarranged overdraft, potentially delaying his access to appropriate advice and support.
Harm: Potential financial and non-financial harm (delayed access to support, risk of suboptimal decisions).
Recommendation: The firm may wish to review its early-stage communication templates for customers approaching financial difficulty, ensuring that signposting to free debt advice is included at the earliest appropriate point. For the customer, a written apology and a summary of available support services could be provided to address the missed opportunity.
At population level: from cases to root causes
The real value emerges when those case-level findings are aggregated. Because every assessment applies the same grading criteria and records drivers and harm categories consistently, Assure 4 can report across the full population of tested cases automatically. Firms can see whether a failure is isolated or recurring, whether it clusters around a particular team, journey or product, and whether it is driven by colleague behaviour or by policy, process and controls.
Reporting is highly configurable, so firms can cut the data by the dimensions that matter to them and align it with their own MI, committee and Board reporting templates. Crucially, every aggregated trend can be traced back to the underlying case-level evidence, so reviewers and risk owners can investigate manually, challenge the findings and satisfy themselves that the root cause has been correctly diagnosed before committing to a population-wide action. And because Assure 4 can re-test consistently against the same criteria, it provides a practical basis for the thematic follow-up reviews that show whether those actions have worked.
Square 4 supports firms across the full outcomes lifecycle, from designing testing frameworks and conducting file reviews to root cause analysis, remediation and thematic follow-up reviews. Our AI-enabled outcomes testing solution, Assure 4, allows firms to test at scale and re-test consistently, so the effect of an action can be measured rather than assumed. If you would like to discuss how we can help, please get in touch.
