How to Measure the Impact of Website Changes

Websites change all the time: a new homepage, a different call to action, a rewritten product page, a faster theme. Months later, someone asks whether the change worked, and nobody really knows.

You don't need complex statistics to answer that question reasonably well. You need a clear goal, a before-and-after comparison done carefully and a record of everything else that happened at the same time.

Step 1: write down what you expect

Before the change goes live, write one sentence:

"We're changing the pricing page layout so more visitors click 'Start free'. We'll compare sign-ups from that page for four weeks before and four weeks after."

This forces three decisions: what you're changing, which number should move, and how long you'll wait. It also protects you from finding a "success" after the fact by picking whichever metric happened to go up.

Step 2: record the baseline

Collect the numbers for the period before the change:

• The main metric (sign-ups from the page, sales, contact forms).
• The traffic to the affected pages, by source.
• The conversion rate of those pages.

Use a period long enough to include normal variation. For most small sites, at least a few weeks.

Step 3: note the date and the details

Write down exactly when the change went live and what it was. "Oct 17, 10:00 – new pricing page, monthly/annual toggle added, FAQ moved up." Include who made it.

This is the step that is easiest to forget and hardest to reconstruct later. WEBPilot's strategy timeline keeps these notes for each website so they're there when you look at the numbers.

Step 4: compare fairly

After the waiting period, compare the same metrics:

• Same length before and after.
• Same days of the week whenever possible.
• Rates, not just totals. If traffic doubled because of a campaign, conversions will rise even if the page got worse. Look at the conversion rate.
• By source. A change might help visitors from ads and hurt visitors from search.

Step 5: check what else happened

This is where most conclusions go wrong. In the same period, did any of these happen?

• A campaign started or stopped.
• A holiday, a sale season, a news event.
• A search engine update.
• Other changes on the site.
• A tracking change, like a new consent banner.

If something big happened at the same time, be honest: the result is uncertain. That's still useful information.

Step 6: decide and record the result

Write down the outcome next to the original note:

• "Sign-up rate from pricing page went from 3.1% to 3.6%. Traffic stable. No campaigns in the period. Keeping the new layout."

The numbers in that example are illustrative; yours will be different. What matters is the format: result, context, decision.

When you can do better: A/B tests

If you have enough traffic, an A/B test is more reliable: half of the visitors see version A, half see version B, at the same time. Because both groups face the same season, campaigns and news, the comparison is much cleaner.

A/B tests need a reasonable number of conversions to give a trustworthy answer. On small sites, they can take a long time. The before-and-after method above, done carefully, is a practical alternative.

Common mistakes

• Changing many things at once. A new design, new prices and a new campaign in the same week make it impossible to know what worked.
• Judging too early. A few days of data are dominated by random variation.
• Ignoring seasonality. Comparing December with November for a retail site tells you about Christmas, not about your page.
• Not writing anything down. Without a record, every review starts from scratch.

Summary

State the goal, record the baseline, note the change, compare fairly, check for other events and record the result. Over time, this record becomes your team's most valuable asset: a list of what actually works on your website. For SEO-specific changes, see How to Measure SEO Results Over Time.