A/B Testing Site Search
With the release of Sajari pipelines, customers can now easily run A/B tests on site search results (A/B testing is included with our Enterprise plans). Catch.com.au has used Sajari pipelines to increase search revenue more than 10%. Considering that about 30% of visitors use a search box and conversion rates involving on-site search can be up to 50% higher than those without, A/B testing site search results is a significant opportunity for businesses to increase revenue and profitability.
Even if you’re not one of the world’s largest e-commerce providers, you can benefit from testing search results. Whether your site goal is...
- Signups
- Downloads
- Fewer support tickets
- Purchases
… or something else, A/B testing your site search can have profound results.
Why A/B test site search?
A lot can influence the effectiveness of search results, including how data is indexed to the order results are displayed to the design of your search results.
What if you gave a small boost to the search results to display the most popular items on your site first? Or what if you gave preference to results related to products or pages your visitors recently rated or viewed?
There are hundreds of permutations on search results that could lead to different outcomes as measured by clicks and conversions.
By tweaking how results are displayed or what order results appear, you can impact search effectiveness, visitor satisfaction, and even website conversions.
What search elements can you test?
You could test the design of your search results or the order they are displayed in. For this article, we’re going to focus on the latter. This includes things such as:
- Product popularity
- Best selling items
- Item ratings
- Brand affinity
Search pipelines for A/B testing
Sajari pipelines are an entirely new way to configure search algorithms with a few lines of YAML. Adding search features such as personalization, range boosting, and spell checking, or search operators such as special filters, can be done with a few lines of configuration.
Pipelines are also the basis for running site search A/B tests. You can run multiple pipelines in parallel and split test them in real-time. For example, half of your searches can be run against pipeline A vs pipeline B.
(Brief aside: Pipelines do not require you to re-index your data or clone your index —pipelines are executed at query time.)
For our example, let’s say we want to test the following hypothesis: We can generate more sales if we boost items that have a higher number of reviews.
To test this, we would create two pipelines: a default pipeline versus one that boosts search results by number of reviews.
A: DEFAULT PIPELINE
B: TEST PIPELINE
Once both pipelines are created you can initiate a search with the new pipeline. The results will automatically show up in the reports. No other configuration required.
Pipelines can be versioned and saved. If you find that your test pipeline is performing better than your default pipeline, you can pause the old one, but it’s always there just in case you need to roll back.
The Sajari dashboard includes metrics such as number of queries and clicks (and any other metrics you decide to track) which will be tracked automatically in your reports. Ultimately, however, what you will want to know is whether one pipeline did a better job of displaying search results that drove revenue.
Conclusion
Search has often been considered a cost center: it’s something you need to spend money on, and unless you’re Google or Amazon, there’s not much you can do with it.
What companies like Catch.com.au have shown is that anyone can transform search into a profit center. Historically this has been very hard because it required expertise and dedicated search engineers. With Sajari pipelines, that’s no longer an issue.
A/B testing is available on Sajari Enterprise plans. Sign up for free to get started with A/B search testing in Sajari today.
More reading on A/B testing and search from around the web:
