Ask an organization whether its automation project paid back and you will usually get a confident answer with nothing behind it.
Sometimes the answer is optimistic, because the project was expensive and everyone would like it to have worked. Sometimes it is unfairly harsh, because the visible remaining pain is easier to recall than the invisible removed pain. Both come from the same root cause: nobody recorded the before state, so there is nothing to compare against.
This is a solvable problem, and it is cheap to solve. It just has to be solved before the project starts, which is the one time nobody is thinking about measurement.
Why the Baseline Disappears
The before state is obvious to everyone while it is happening, which is exactly why nobody writes it down.
Then the automation ships. Within a few weeks the manual process has faded, people have reorganized their days, and the honest answer to "how long did that used to take" becomes a shrug and a guess. The guess is usually wrong in whichever direction suits the speaker.
A baseline captured in twenty minutes beats a reconstruction attempted six months later, every time.
What to Measure Before
Five numbers. None requires a project to collect.
1. Repetitions per Period
Count them. Do not estimate them.
This is the single most important number, because automation pays back per repetition. It is also the one most often wrong from memory, usually inflated by the memory of the pain.
2. Minutes per Repetition
Time the person doing the work, once, with a timer. Not their estimate. People are reliably poor at estimating durations of tasks they find tedious.
3. Share of Time the Process Consumes
The proportion of a role or team's capacity the process eats. This is the number executives respond to, because it translates directly into what the team is not doing.
When we started with Class Action Settlement House, this number was the whole story:
The time spent loading the cases is 80% to 85% of the time we are spending.
Lee, on the manual process
That figure made the business case before anyone estimated a build. It is also what made the result measurable afterward: roughly 13,500 hours of manual processing eliminated and about $450,000 saved.
4. Exception Rate
What share of cases do not follow the normal path. Ask the person who does the work, then add ten points, because they are underestimating.
This number determines whether the project is viable at all, and it also sets expectations for what "done" will look like.
5. Error Rate and How Errors Surface
How often the manual process gets something wrong, and whether the mistake is caught the same day, in a week, or by a customer. Automation usually changes the error profile rather than eliminating errors, and you cannot tell whether that change was good without knowing where you started.
What to Measure After, and When
Not in week one. The first weeks after launch are unrepresentative: people are still verifying output, edge cases are surfacing, and the process is being adjusted.
Wait until the process has run enough cycles to be routine, then measure:
- The same five numbers, collected the same way.
- Verification time, which is new and did not exist before. Somebody is checking the output, and that cost is real even though it declines.
- Cycle time, meaning how long a single item takes from arrival to finished. This often improves more dramatically than hours saved, and it is what internal customers actually notice.
- What the reclaimed time went to. If nobody can say, the saving may not have been captured.
Hours saved that were reabsorbed into slack are a real quality-of-life improvement and not a financial return. Both are legitimate outcomes. They are not the same outcome, and conflating them is how organizations end up disappointed by projects that worked.
The Number That Matters Most
Net hours saved, not gross:
(repetitions × minutes saved per repetition) − verification time − exception handling time − maintenance time
An automation that saves twelve hours a month and costs three hours a month to check and maintain is saving nine. That is still a good project. It is a worse project than the proposal implied, and the gap between those two figures is where the disappointment lives.
The full cost picture is in What an Automation Project Actually Costs.
Metrics That Mislead
Some numbers look like evidence and are not.
- Percentage automated, without volume. Automating 100% of a quarterly process is worth less than automating 60% of a daily one.
- Hours saved, gross. Always ask what it costs to run.
- Tickets closed or items processed, without cycle time. Throughput can rise while individual items get slower.
- User satisfaction immediately after launch. Measures novelty, or change fatigue, more than value.
- Anything self-reported months later. This is the reconstruction problem again.
Good After-Metrics Look Specific
The useful pattern is a small number of concrete operational facts rather than a broad claim of efficiency.
For SIMA, a membership association, members had been keeping a separate account for every tool, and onboarding a new member meant manual setup in each system separately. After we implemented single sign-on across their connected tools, the measurable facts were narrow and checkable: roughly 200 single sign-on logins per week, and new members provisioned across every connected system in about two minutes after joining.
Neither of those is a sweeping efficiency claim. Both are verifiable, and both describe something that was previously slow in a way anyone in the organization can confirm.
A good after-metric survives someone asking "how do you know."
When the Numbers Say It Did Not Work
Sometimes the honest measurement shows a marginal result. This is useful, not embarrassing.
A marginal result usually means one of four things: the volume was lower than believed, the exception rate was higher than believed, the verification cost has not declined as expected, or the saved time was reabsorbed rather than redirected.
Each has a different fix, and you can only tell them apart with a baseline. Organizations that measure honestly get better at choosing the next project. Organizations that do not measure repeat whatever they did last time.
A Simple Rule of Thumb
Spend twenty minutes measuring the week before the project starts.
Count the repetitions, time one instance, and write down what share of the day it eats. That is enough. Everything else can be reconstructed from those three numbers, and none of it can be reconstructed without them.
Final Thoughts
Most organizations cannot say whether their automation paid back because the before state was obvious at the time and nobody wrote it down.
Do the measuring and the next automation decision gets easier, because you will be arguing from evidence instead of from memory.
If you want help establishing a baseline before an automation project, or an honest read on whether the last one paid back, we are happy to work through it with you. Book a short consult.