SEO Split Test Result: Does Adding the Category Name to Product Page Titles Make a Difference?

Brian Moseley

Dec 17, 20213 min read

Before you start: what do you know about SEO split-testing? If you’re unfamiliar with the principles of statistical SEO split-testing and how SplitSignal works, we're suggesting you start here or request a demo of SplitSignal. 

First, we asked our Twitter followers to vote:

Only 6% of our followers guessed it right, the test result was negative.

Read the full case study to find out why.

The Case Study

Writing relevant page titles is still essential for SEO. Not only because it gives Google a good idea of ​​the topic of the page and the queries for which the page should rank, but also because the page title is often the most prominent element a user sees in the search results. In search, you are always competing with 9 other search results; this is why it is essential to write the most relevant page title.

Product pages on an ecommerce website are often the most important pages for monetization. Running tests on these types of pages to find the best possible answer for both search engines and users can be very insightful when it comes to optimizing your organic performance. For a large ecommerce company in the Netherlands, we added the product category name to their product page titles to see if it made their search results more relevant and effective.

The Hypothesis

The website in question used the pages’ H1, which is the product name + the brand name of the company.

We hypothesized that adding the category name would add additional relevance because it tells users and search engines something about what the product is. We also thought it would increase our visibility. By adding the category name, we also add (relevant) keywords for which we want to rank in combination with the product name itself. 

We wanted to validate whether adding the product category name to the product page titles would have a positive effect on organic traffic, so that’s what we did.

The Test

We used SplitSignal to set up and analyze the test. We used a larger sample size for this test. About 6,500 product pages were selected as either variant or control. To ensure changes to SERPs are seen, SplitSignal also measures whether Googlebot has seen the variant pages since launch. As mentioned, for this test, we added the product category name to the page titles of the variant:

The Results

We ran the test with SplitSignal for 21 days and saw the following results:

Almost immediately after the test started, traffic from our modelled control group outperformed the actual traffic from the variant group. This means that the traffic to the variant group performs worse than predicted and that the result of the test is negative. 

At the end of the test, we had a relative result of -7.1%. The cumulative view shows not only that the change has a negative effect on clicks but also that we performed a significant test. When all three curves perform below (negative) or above (positive) the y=0 (cumulative gradient) axis, the test is statistically significant. At the end of the test, the confidence level was 99%. This means we can be 99% sure that adding the product category name to the page title will negatively affect the organic traffic to the template.

As mentioned at the beginning, page titles are still essential. They clearly play a role when it comes to driving organic traffic. Relevant page titles will be effective for both search engines and users. If you don't get it right, you can lose traffic, as this case shows.

Looking at the data, we saw not only a drop in clicks but also a 4% drop in impressions compared to our modelled control group. This is a telltale sign that our search rankings have declined. For both search engines and users, this was no better answer to their query than the original page titles.

In retrospect, we can conclude that the page titles were descriptive enough for Google to know for which search queries these pages should rank. In addition, we saw that many page titles became too long, resulting in truncated results. Since the website in question is a well-known brand in the Netherlands, this may have contributed to the negative result of the test. Users are known to scan search results; if the information they may be looking for is missing, the result is less relevant. We can conclude that the quality and readability of the results is an important factor. 

User-oriented tests are often the most difficult to predict. Sometimes you are convinced that a certain optimization will win. This test is a perfect example of why you should validate changes before deploying them to your website. Testing can be a great ‘safety net’ when it comes to changes that negatively affect your organic performance. For the website in question, it’s back to the drawing board to explore which changes are having a positive effect on their organic performance.

Sign up for our newsletter with the most recent case studies via the request form.

Have your next SEO split-test analyzed by OrangeValley Agency.