Killing experiments: the discipline most teams lack.

The discipline most marketing teams lack is not creativity. It is the willingness to kill an experiment on schedule and act on what it told you. Starting tests is easy. Ending them — declaring a winner, a loser, or a no-result, then moving on — is where most teams fall apart.

I have run growth programs where the backlog held forty live experiments and not one of them had a kill date. Everyone was busy. Nobody was learning. The experiments had become a to-do list that never shrank, because nothing ever got closed. A good experiment program is measured by how fast it ends tests, not how many it starts.

Why do teams struggle to end experiments?

Teams struggle to end experiments because ending one forces a verdict, and verdicts feel risky. A live test is comfortable. It is proof that you are trying something. The moment you call it, you have to admit it worked or it did not — and admitting a test failed feels like admitting you failed.

That is the first problem. Ego gets attached to the hypothesis. Someone championed the new landing page, the new headline, the new offer. Killing it reads as a personal loss rather than a result. Good programs separate the person from the test. The hypothesis is a question, not a reputation.

The second problem is softer and more common. Nobody owns the end. Someone owns the build. Someone owns the launch. The ending has no name next to it, so it drifts. A test that was supposed to run two weeks runs two months because checking it, reading it, and closing it never made it onto a single person's plate.

The third problem is the sunk-cost trap. You spent three weeks building the variant. Calling it a loser means that time produced nothing — or so it feels. In truth, a clean negative result is worth exactly what it cost. It tells you where not to spend the next three weeks. That is not nothing. That is the product.

What does it cost to leave experiments running?

Leaving experiments running costs you more than the test itself — it costs you the decisions you could have made with the answer. Every open experiment holds a piece of your traffic, your budget, and your attention hostage. An unresolved test is a decision you are refusing to make.

Start with traffic. A San Diego dental practice running an A/B test on its booking page is splitting every visitor between two versions. If one version converts worse and you let it run six weeks past its call date, you paid for every one of those lost bookings. The math is real. Say the page gets 800 visitors a month and the losing variant converts two points lower. That is sixteen lost booking requests a month, bleeding out while nobody closes the test.

Next, budget. Paid experiments that nobody ends keep spending. I have seen an ad set marked "test" still running nine months later, at $1,200 a month, because it was never formally killed. Nobody remembered the hypothesis. The spend had become furniture.

Then attention. This one is harder to see and more expensive than the rest. Every open test is a tab in someone's head. A team carrying thirty undecided experiments is carrying thirty unfinished thoughts. They cannot plan the next round because the current round never ended. The backlog stops being a plan and becomes a monument to indecision.

How do you set a kill date before you start?

You set a kill date by writing it into the experiment before a single line of code or copy is built. The kill date is not a nice-to-have. It is part of the hypothesis. No date, no experiment.

Here is the minimum you write down before anything goes live:

1. The hypothesis. One sentence. "We believe moving the phone number above the fold will lift calls, because mobile visitors want to tap, not read."
2. The one metric that decides it. Not five metrics. One. Calls per visit. If you cannot name the single number, you are not ready to run it.
3. The threshold. What result counts as a win. "A 15% lift in calls per visit, or better."
4. The sample or the date. Either "we stop at 2,000 visitors per variant" or "we stop in 21 days," whichever comes first. Pick the harder of the two for low-traffic sites so you do not call a winner off twelve conversions.
5. The owner of the ending. A name. The person who reads the result and writes the verdict.

That last line is the one most teams skip, and it is the one that decides whether the test ever ends. A kill date with no owner is a suggestion. A kill date with a name is a commitment.

For low-traffic professional-service sites — most of our clients — the date matters more than the sample, because the sample may never arrive. A family-law firm in San Diego might get 400 relevant visitors a month. You will not hit statistical significance on a small headline change in two weeks. So you set a date, read the direction, and make a judgment call. That is honest. Pretending you have significance you do not have is not.

How do you read a result without fooling yourself?

You read a result by comparing it to the threshold you wrote down before you started, and nothing else. The threshold is your defense against the story your brain wants to tell. Humans are good at finding a win in any data if they are allowed to move the goalposts.

There are three honest verdicts. A test won. A test lost. A test produced no clear signal. The third one is the most common and the least respected. A no-signal result means the change did not move the metric enough to tell, either way. That is a real answer. It tells you the lever you pulled is weak, and you should pull a bigger one next time.

Watch for the traps. Do not peek early and call it when the numbers look good on day three — early leads reverse constantly. Do not slice the data ten ways until you find a segment that won; if you test enough segments, one will look positive by chance. Do not keep a losing test alive "just a little longer" hoping it turns. If it is losing against your threshold by the kill date, it lost.

The hardest reads are on small sites. When the numbers are too thin for significance, say so out loud. Write "directional, not conclusive" in the verdict. Then decide based on direction plus judgment — the cost to ship, the risk if you are wrong, the size of the apparent effect. A clear verdict that admits its own uncertainty beats a confident verdict built on nothing. This is a large part of what a Fractional CMO brings to a small team: the authority to call a test dead and the honesty to say when the data cannot decide.

What happens after you kill an experiment?

After you kill an experiment, you record the verdict, ship the winner or revert the loser, and write one line of learning into a place the whole team can see. The kill is not the end. It is the handoff to the next decision.

Three things happen in order. First, the verdict goes in writing — won, lost, or no signal, against the threshold, with the date. Second, you act. A winner gets shipped to all traffic and the old version retired. A loser gets reverted the same day, not left split "for now." Third, the learning gets logged. One sentence. "Moving the phone number above the fold did nothing for a desktop-heavy audience — skip this pattern for professional-service sites."

That log is the asset. Run a program for a year and you build a record of what works for your market, your traffic, your clients. The next experiment starts smarter because the last one ended cleanly. Teams that never close tests never build this. They repeat the same hypotheses because nobody wrote down that they already failed.

We saw this discipline pay off with McShanes Solicitors. The gains did not come from one clever idea. They came from running tests with dates, reading them straight, and killing the ones that did not earn their place — then pouring the freed-up budget and attention into the handful that did.

This is also why a running program beats a one-time project. If you are weighing whether to bring this discipline in-house or hire it, read when you actually need a Fractional CMO (and when you don't) — the answer often turns on whether anyone currently owns the ending of things. And if you are deciding between an outside operator and a vendor, Fractional CMO vs agency: the difference that matters covers who stays in the room when a test needs calling.

Where this breaks down

This discipline breaks down when you have too little traffic to learn from anything, no matter how clean your process is. A solo consultant with 150 site visits a month should not be running A/B tests — they should be fixing the obvious and talking to clients. Experiment discipline only earns its keep once you have enough volume to read a signal. Below that, the honest move is to stop testing and start shipping the clear wins. Do not dress up guessing as a program.

Killing experiments is not a sign that the program failed. It is the program working. The teams that learn fastest are the ones most willing to end things on schedule, write down what happened, and move to the next question. Starting is easy. Ending well is the discipline.