4 Best Practice Tips for Crafting KPIs to Drive Successful Innovation Trials

Nick Blamire Brown, Senior Consultant, Isle UK

This blog is the last of a 3-part series unpicking what makes the Trial Reservoir model – boasting a 70% success rate – so distinctive. In it, we explore the elements that matter most when designing tech trials. If you missed the beginning, jump back and read the first post here.

Innovation trials in the water sector often brim with potential, but many never progress beyond the trial phase. The culprit? Vague or misaligned key performance indicators (KPIs) that leave decision-makers unsure whether a solution really worked. Without clear, measurable outcomes, promising technologies get stuck in the “piloting valley of death.”

In this blog, we explore the most common challenges to implementing effective KPIs, and highlight some best practices that can help you design KPIs to optimize trial success.

The Challenge

Water utilities and solution providers alike struggle when trial KPIs are inadequate. In practice, an innovation trial with muddled KPIs is a trial set up to fail. Specific challenges include:

• No clear “definition of success”: If the trial’s goals aren’t quantified, stakeholders may have different perceptions of success. For instance, a utility’s operations team might expect a new filtration technology to cut chemical costs by 20%, while the vendor is thinking success means “water looks cleaner.” At the end of the trial, without predefined metrics, it’s impossible to decisively say whether the technology met expectations. This often leads to “let’s do another pilot” instead of a purchase.
• Delayed decision-making: Utilities are often governed by committees and cautious leadership. When trial results come back without clear alignment with KPIs, decisions get deferred. Water utilities frequently conduct pilots that rarely lead to a sale or full-scale implementation because there’s no guarantee of what happens even if a trial seems successful. The result is a lot of time and money spent trialling, with little to show in terms of actual adoption.
• Stakeholder misalignment: Water sector trials typically have multiple stakeholders — R&D teams, operations staff, design engineers, finance & procurement teams. Without a common set of KPIs co-created at the trial’s start, each stakeholder may chase a different outcome. This can cause friction (“The lab says water quality improved, but the budget holder says the ROI wasn’t proven”) and ultimately undermines confidence in the trial outcomes.
  

Why it matters

Better KPI design isn’t just a bureaucratic exercise — it’s the key to unlocking faster rollouts and greater return on investment (ROI) for all parties involved. When trial KPIs are well-crafted, everyone wins.

• Faster path from pilot to implementation: A trial with clear success criteria creates a built-in decision point. If the technology hits the numbers, there’s no excuse for delay — the roadmap to scale-up is ready. This clearly-defined milestone can dramatically accelerate implementation timelines, moving a project from testing to full deployment in months instead of years. Our experience shows that when pre-agreed KPIs are met, the end-user is obliged (and generally eager) to move forward. In other words, clear KPIs act like a green light for adoption.
• Stronger ROI and business case: Utilities operate under tight budgets and regulatory scrutiny. They need solid evidence to justify investing in a new solution. Objective KPIs translate data-driven trial results into the values-based language of ROI, efficiency, compliance, or customer service. For example, a filter trial that shows that a technology “met all water quality standards and reduced energy costs by 15%” provides a concrete business case for procurement. This not only helps operational managers secure internal approval, but it also gives technology vendors the hard proof they need to market their solution elsewhere.
• Increased technology uptake across the sector: Zooming out, when one utility confidently implements a new technology after a KPI-driven trial, other utilities take notice: success breeds success. Clear KPI frameworks and published results (when shared in industry forums or case studies) give the wider water community reference points for what “good” looks like. Over time, this leads to greater overall uptake of innovation as the industry develops its collective capability for trialling and adopting new solutions. 

A model for KPI-driven trials as a roadmap for implementation

Isle Utilities created the Trial Reservoirs Initiative as a novel funding and support mechanism to accelerate water sector innovation, and its structure effectively bakes in best practices for trial KPIs. 

In simple terms, Trial Reservoirs Initiative provides funding to technology companies to run a full-scale trial at a utility site, under the agreement that if the trial meets its pre-agreed KPIs, the utility will purchase the technology. If the trial fails to meet the KPIs, the utility isn’t obliged to buy, and the tech company doesn’t have to repay the funding – essentially making the trial “free if it fails”. 

This win-win financing model removes financial risk, but just as importantly, it enforces discipline in trial design. Before any Trial Reservoir project kicks off, Isle’s team brings the end-user and vendor together to define clear trial KPIs and success factors.

This means no trial goes forward without both sides aligning on what success looks like. Crucially, the KPIs set in Trial Reservoirs Initiatives are not academic or trivial metrics — they are directly tied to the end-user’s go/no-go implementation decision. These indicators are objective, measurable, and meaningful to the utility’s operations: the entire trial is structured around the question “Can I implement this?”

Isle Utilities ensures the KPIs are mutually accepted and that they will provide all the data the end-user needs to confidently answer that implementation question. By requiring a Trial and Purchase Agreement upfront, Trial Reservoirs Initiative cements the link between KPI success and implementation. In fact, the contract between the utility and the vendor includes the agreed KPIs and the commitment to proceed if those KPIs are met. 

How to do it better

Drawing on the Trial Reservoirs Initiative international experience and lessons learned, here are four best-practice tips for crafting KPIs that drive successful trials:

1. Co-develop KPIs with end-users and technology providers

Don’t let KPIs be dictated by one side alone. Instead, bring both the end-user and the solution provider to the table when defining trial goals. This co-development ensures the metrics matter to the end-user and are achievable by the technology. It also builds joint ownership of the trial outcome. 

In the Trial Reservoirs Initiative, Isle’s experts facilitate this process, resulting in trials with meaningful, measurable metrics of success that are agreeable for both parties. For example, the Trial Reservoir trial between EnBiorganic and Paper Excellence was driven by Paper Excellence’s Zero Liquid Discharge (ZLD) policy. Isle worked with Paper Excellence to scope two clearly-defined KPIs around the wastewater’s aesthetics (i.e., no odour or colour in the treated water) and an absolute value threshold chemical oxygen demand that aligned with this broader strategic goal. These metrics also meant both parties knew what success would look like. The trial exceeded expectations, meeting all its KPIs – a success which has led to a 3-year service agreement.

KPI co-development allows end-user and vendor to  reach a mutual understanding before the trial begins, which dramatically increases the chances that a successful trial will be recognized as such by all involved.

Key takeaways:

• Involve both the utility and solution provider in setting trial goals.
• Ensure KPIs reflect what the end-user truly values and what the technology can deliver.
• Co-creation builds ownership and mutual accountability for results.
• Avoids mismatched expectations and helps ensure everyone agrees on what success looks like.

2.Ensure KPIs are objective and measurable

It’s surprisingly easy for trials to start with squishy goals such as “make the process easier” or “improve water quality”. 

Best practice is to translate broad objectives into specific, measurable, achievable, realistic, and timely KPIs.  The KPI should be based on factual data, not opinion – anyone reviewing the results should come to the same conclusion. It should have a well-defined method to get the data. And the KPI should be reasonably achievable over the duration, within a specific time period. If an objective cannot meet these criteria, it shouldn’t be a KPI.      

During Bynry’s Reservoir trial with Samoa Water Authority (SWA), Isle helped Bynry to refine the KPIs around the implementation of their digital water Smart360 platform. In collaboration with experts from the Asia Development Bank, Isle worked with both Bynry and SWA to develop a set of measurable, actionable, and time-bound metrics that set bounds around whether deployment of a given module would be considered “successful”. These metrics played a major role in the pilot’s success, and resulted in the SWA committing to roll out the Smart360 platform across their entire network.

By pinning down things like units, measurement frequency, and data sources upfront, you eliminate ambiguity. This also deters the temptation to retroactively change goals. Clear KPIs act as the north star during the trial – the whole team knows what they’re aiming for, and come evaluation time, there’s no debate about whether the target was reached.

Key takeaways:

• Translate vague goals (e.g., “better quality”) into precise, quantifiable metrics.
• Use factual, verifiable data points to eliminate interpretation disputes
• Define how data will be collected, including frequency, method, and source.
• Set realistic performance thresholds that can be achieved within the trial timeframe.

3. Use KPIs to define a monitoring plan with clear ownership

A KPI on paper means little if there isn’t a plan to monitor it during the trial. 

Treat the KPI list as the foundation of your trial monitoring plan. For each KPI, define how and when it will be measured, and who is responsible for tracking it. By assigning clear ownership, you ensure that data collection doesn’t fall through the cracks. This also spreads involvement among stakeholders, so that everyone is engaged with the trial.     

As part of their Trial Reservoir pilot with Yorkshire Water, de-watering specialists Orège UK Ltd  clearly defined roles and responsibilities for monitoring during and after the trial. The monitoring plan detailed who was responsible for measuring each aspect, the methods to be used, and the frequency and channel for reporting.  This approach helped to evaluate the technology’s efficiency and operational feasibility under specific performance criteria, with each party clear on the variables that led to success. 

In practice, a good monitoring plan might be a simple table listing each KPI, its measurement method, data source, responsible person, and reporting frequency. Regular check-ins can then be held to review interim data. This level of organisation not only keeps the trial on track but also boosts confidence in the results (since everyone knows the numbers have been diligently collected and vetted). When the trial concludes, the team can present a well-documented KPI report to decision-makers, making the go/no-go decision much more straightforward.

Key takeaways:

• Assign responsibility for tracking each KPI to specific team members.
• Build a simple monitoring plan covering what to measure, how, and when. 
• Maintain regular check-ins to review progress and ensure data integrity.
• A clear monitoring framework builds trust in the trial outcomes and speeds up decision-making.

4.Make KPIs meaningful and tied to real end-user value

Every KPI should tie back to something the end-user truly cares about. In the water sector, that often means linking to value drivers like cost, compliance, customer service, resilience, or sustainability. The question to ask is: “If this KPI is met, will the end-user see a clear benefit?” 

Tying KPIs to end-user value also means aligning with the end-user’s broader strategic goals. This alignment makes it much easier for utility executives to champion the project. A good test is to imagine explaining the trial results to a CEO — would the KPIs you picked resonate with them? If so, you’ve likely made them meaningful. 

KPIs that are technical for the sake of it can lead to hollow “successes” that don’t persuade anyone. The Trial Reservoirs Initiative emphasizes the importance of KPIs that answer the question “Can I implement this?” – essentially de-risking the implementation from the end-user’s perspective.

Key takeaways:

• Every KPI should connect directly to a value driver (e.g., cost, compliance, sustainability).
• KPIs should reflect outcomes that matter to decision-makers, from operators to executives.
• Test your KPI choices by asking: “Would this resonate with a CEO?”
• Avoid overly technical metrics that don’t translate into clear business impact.

In the water sector, where innovation is essential, but adoption is often slow, getting trial KPIs right isn’t just helpful — it’s critical. Robust, clearly defined KPIs are the foundation for any successful trial, turning what might otherwise be an open-ended experiment into a focused effort to establish a measurable path to implementation and benefit realisation. They bring clarity, build trust, and enable faster, more confident decisions. By following the best practices outlined here, organisations can ensure that the time, effort, and resources invested in trials lead to meaningful outcomes.

Want to run more effective innovation trials? Isle can help you design KPI-led pilots that drive real implementation. Contact Jo Burgess ([email protected]) to explore our Trial Reservoirs Initiative.