What is site search?

Site search (also called on-site search) is a search engine that is confined to a single site, connected sites, or domains. For example, you may want to only search a specific site, or you may have multiple subdomains and a parent www site to search across.

Why not use Google for site search?

Google is the undisputed heavyweight for internet search, but site search is a different beast. In fact, Google retired its Google Search Appliance and Google Site Search products years ago.

Google, Bing, DuckDuckGo and other search engines are ideal for driving visitors to your site, but for providing a great on-site experience, a site search engine is required.

Unlike general internet search engines, site search engines can do things like:

• Provide personalized search

• Be optimized to outcomes (such as conversions or sales)

• Display custom related results and special promotions

• Be A/B tested to improve search results or relevant product results

• Offer much richer user interfaces

• Have deeper search analytics to understand visitor intent

How important is site search?

Here are just a few interesting facts about the importance of good site search:

• Around 40% of online transactions involve an on-site search, particularly for e-commerce websites, ie, your visitors want to use the search bar on your site.

• Buyers are 90% more likely to have used your site search during a visit and can account for 45% of site revenue.

• Now that more than 50% of web traffic is mobile, mobile-optimized site search is vital. In fact, 50% of total e-commerce revenue comes from mobile and 79% of users restart their Google search if the page they land on isn’t mobile-friendly.

• When one company made their on-site search more visible...
  - Site searches climbed by 439%
  - Visitors stayed on the site 110% longer when they used search
  - Page views increased by 45%
  - Goal completions increased by 242%.

• Sites with a semantic-based search engine have a low 2% rate in shopping cart abandonment, compared to as much as 40% on sites with plain text search.

• It's important to optimize search for mobile and desktop — mobile users put items into their shopping carts at almost the same rate as desktop users.

• Speed matters. Page load time affects the purchase decision of almost 70% of shoppers. Amazon found that at 100 millisecond delay can cost millions of dollars.

• Search design is incredibly important. Baymard Institute noted that 57% of e-commerce searchers weren’t sure if they got the results they were looking for.

Who is this buyer's guide for?

Site search is a broad term that could refer to website search, on-site e-commerce search, intranet search, and more. For this guide, we are limiting an overview to features primarily for the following three use cases:

Site Search‍

• Visitors to your public site

• Search across one or more domains

• Static or CMS-based websites

E-commerce Search‍

• Search for online products and related metadata (price, color, size, etc.)

• Search category pages and collections

• Platform (e.g., Shopify, Magento) or custom stores

App Search‍

• Web or mobile-based application search

• Typically SQL or NoSQL database-managed content

What should I look for in a site search solution?

With many open source and hosted solutions to choose from, which site search platform is right for you? We explain the factors that go into selecting a site search product below.

Site search engine features

What happens as users type in a query on your site search today? Do they get auto-suggestions (like Google and Amazon’s search) or do they get more visual results (like IMDB or AnnTaylor.com search)?

Does your site search know how to handle typos or misspelled words? Does it search only web pages (HTML) or does it also index documents (PDF, DOC, DOCX, etc)?

Site search user experience experience (UX) can vary a lot. By default, most content management systems (CMS) and legacy solutions offer very limited functionality and poor user experience.

Today there are many features for site search to build a great end-user experience for your audience. Let's look at some of the most important ones.

Crawlers and API indexing

How your site is indexed is important. Most sites can be indexed through search crawlers, but mobile apps or e-commerce sites typically require APIs to connect to SQL and NoSQL data stores. Without getting into all the detail, some questions to ask include: how quickly will this solution index my site? When an edit or change is made, how quickly is that re-indexed? If a new page is added, how does that get added to the index. If the site search solutions that you’re evaluating cannot provide great answers to those questions, it may be time to look elsewhere.

Connect to business data

What if your site search engine could connect outcomes like signups, conversions, and sales to the search queries that led to that result? Google optimizes ad results in precisely this fashion — displaying ads with the highest conversion rates. It’s available for site search as well, but to do that your search provider needs to be able to connect to business data (ideally in real time) to continually optimize results based on user behavior.

Spell checking

Nearly 1 in 10 search queries are misspelled. That’s a lot of misspelling! Your site search tool needs to know how to handle misspellings or many of your customers will leave your site mistakenly believing you don't offer what they need. Ideally, a search engine should include typo tolerance — a “did you mean?” feature, fuzzy search spell checking, and/or at least display multiple possible results based on similar keyword searches.

Autosuggest

In old school search, users type in a query and hit return to see results. Consumers, however, expect at minimum a Google or Amazon-like search with suggestions, or autocomplete or autosuggest, as they type. Dynamic search suggestions not only provide instant gratification, but they also help sort out misspellings faster (see “spell checking” above).

Synonyms

You say “sneakers,” I say “running shoes.” Your users are typing in search terms that may be different from how your site is optimized. At a minimum, a modern search engine should support synonyms and/or "fuzzy search”. Newer search engines are starting to use vectors, too, a mathematical approach to understanding language that will eventually supersede the need to build synonym libraries. However, until vectors are broadly available, your site search should support a synonym library.

Instant search

Search suggestions (see above) are a very common search UI feature. Now many companies are using using instant search on their site where search results change in real time as users type in their queries. Instant search works well for more “visual” product sites. It may be worth A/B testing your site search (more on that below) to see what works best for your CTR.

Analytics

How do you know if your site visitors are finding what they need? Your site search solution needs to have metrics — or at least be able to work with 3rd party solutions like Google Analytics or another business intelligence solution — so you can analyze how users are interacting with search results. This can help you to determine whether certain results need to be boosted, how search trends are changing over time, which queries are returning poor results, and more.

A/B testing

What if, for a given query, you changed results? Would you have better or worse click-throughs, conversions, sales, or user satisfaction? A/B testing can help. Tests could be performed on everything from search terms to how your data has been indexed to the search results design. A modern search solution should feature search A/B testing and provide guidance on what search algorithm helps your company improve its bottom-line results based on whatever criteria you’ve established.

Rules

Your site search should have a way to manage how search results are ranked. For example, you may want to exclude documents from being indexed. Or you may want to boost specific products or certain types of content (e.g., blogs) and lower ranking for other types of content (e.g., comments). These are powerful ways to impart business knowledge into your search results.

Speed

Speed is an invisible feature, but incredibly important. Amazon showed that just .001 second differences in returning results meant huge losses. It’s not just true for online stores. Your users are accustomed to Google-like speed. Anything less is likely to send them elsewhere. Your search tool or search provider should be clocking-in in milliseconds.

Document indexing

Documents (.doc, .docs, .pdf) available for download on your site — whether they’re gated or not — should be indexable as well. If you don’t want them indexed, that’s something you should be able to manage through rules (more on that below!).

AI and machine learning

If your search provider doesn’t support machine learning algorithm ranking out of the box, look elsewhere. It’s not a nice-to-have anymore. It’s a must-have, especially for sites with large indexes. Search engines can use reinforcement learning to score search results and user behavior to automatically improve results over time. A smart search engine should know when a user converts (signups, shopping cart, sale, download, or something else) on a search to boost results like that for future queries.

Personalization

Built-in personalization features and/or connecting to third-party personalization solutions should be on the list as well. Search personalization creates contextual profiles for  individual visitors to personalize their search results and display relevant content. Personalize results based on user preferences, location, gender, past purchase history, product type, and more.

Filters and facets

Allowing your customers to filter results by price, content type, author, or other factors is very useful especially for sites that have hundreds or thousands of records. Your search platform should support both search facets and filters to help customers narrow down results to find exactly what they need.

Design and SDK

We touched on other UX/UI elements above such as instant search, facets and filters, and search suggestions. There’s also the question of how to get search to match your site design. This includes both designing the search bar as well as the search results pages. The search solution you choose should ideally allow for a simple UI builder to match colors and CSS styles. For more sophisticated styling, design libraries (or SDK toolkits) for popular languages like ReactJS and Javascript libraries should be available.

Multi-language

Does your search engine support Latin script languages (think English, French, Spanish, etc) and multi-byte symbol-based queries (Chinese, Japanese, etc.)? Moreover, are the machine learning models your search provider includes multilingual? You may not need it today, but when you do, you’ll want a search engine that can scale to meet the needs of new audiences.

Scaling site search

Companies like Amazon, Reddit, and Aliexpress can afford to hire full-time search engineers and system reliability engineers (SREs) to run their site search (Amazon alone has over 1000 people in their search team!). Most organizations don’t have the luxury to hire a battalion of engineers. Many others would prefer to leverage their engineering talent in other parts of the business.

There’s a spectrum of needs and where your company and use case falls can largely help make this decision. Every use case is different, so take this only as a way to frame thinking.

• Company website: Site search should almost always be hosted for the simple reason that the data is flat, easy to index, and not private.

• Online store: Unless you’re selling highly illicit material or have specific hosting requirements, hosted search will likely meet your needs.

• Sensitive data application: Financial applications or other regulated or highly sensitive data may benefit from self-hosting.

• Internal KB: For corporate wikis or knowledge bases, it’s valuable to have information that’s on-prem or on the public internet secured through SSO.

SaaS vs on-prem: which is best?

Which is best for site search?

Management

✅ SaaS: There’s nothing for you to manage. The service is fully managed so you can sleep better at night.
‍
❌ On-Prem: You are responsible for upkeep. One or more engineers or IT staff required. For search that directly drives revenue (e.g. ecommerce product pages), this also means you need people on call 24-7.
‍

Scaling

✅ SaaS: Your provider should be set up on elastic infrastructure that can expand or contract to meet demand.
‍
❌ On-Prem: This can be set up on elastic infrastructure and you’ll need to size servers and CPUs accordingly to manage data I/O. There’s more management overhead, but also more control.
‍

Maintenance and SLA

✅ SaaS: Hosting company should have a service-level agreement to manage upgrades or outages with an agreed-upon time frame.
‍
❌ On-Prem: Internal SLA plus SLA from your IaaS or PaaS provider.  
‍

Security

🟨 SaaS: Depending on the security and privacy level required, many SaaS companies encrypt data (both in transit and at rest) and offer SSO and other security tooling.
‍
🟨 On-Prem: You have a lot more control over how you want to secure your service, but there’s extra costs to do so.
‍

Pricing

✅ SaaS: All-inclusive pricing.
‍
❌ On-Prem: Infrastructure costs will be smaller, but that is the only thing and this is arguably the smallest of all costs. Engineering, devops, monitoring, staff on-call, etc. will add up fast.  
‍

Upgrades

✅ SaaS: Generally speaking, this is where SaaS software shines. Your search provider manages upgrades and improvements.
‍
❌ On-Prem: Your team will need to upgrade software on a scheduled time slot. For many businesses, this happens only once every 6 to 12 months.  
‍

Uptime

🟨 SaaS: Managed for you and should be ready to scale with high availability (HA) and failover.
‍
🟨 On-Prem: With a large enough team, you should be able to manage uptime and build high availability and failover into your architecture. Search is generally equal to or more complex than database management, so typically you should allocate resources as you would with DBAs.
‍

Community

❌ SaaS: If the hosting provider has a community, it can be incredibly valuable especially for new customers.
‍
✅ On-Prem: If you’re using an open source solution, this is where it can really shine. Popular OSS projects have large communities that can provide support, best practices, and troubleshooting.  
‍

Open source search software

Related to the question above of hosting or going on-prem is whether the solution should be open source or not. If you choose to use a free open source search engine it typically means you’ll be going on-prem (see above). (To add to your choices, there are open source options, most notably Elastic, that offer SaaS services as well. But beware, many open source companies are rethinking their licenses.)  

For some companies who want to use open source and have the capacity to manage it in-house, this is a small price to pay to support open source development. We love open source ourselves and have open sourced many modules.

For many businesses, however, the complexity of managing an open source search engine on premises, the need for high-availability, and the desire to receive enterprise-level support outweigh the value of using free open source software (FOSS) for site search. This is a business question and one that’s worth asking.

Pricing

Most site search is priced on the number of monthly queries and/or the total number of records stored in the search index. There are a great many other features that may be marked up (for example, some search providers require a premium payment for machine learning). Questions you will want to answer before selecting a provider:

• How many searches do we anticipate we will have each month?

• How many objects or records need to be indexed?

• Does my provider charge extra for any basic or advanced features like crawling, A/B testing and re-indexing, AI and machine learning, personalization, etc.

• Is set up and onboarding included or separate?

• What other services do you offer?

• What is included or excluded in our set up fee?

Ease of use for technical and non-technical users

Who will be managing your site search software? Will it be developers only, or will your marketers, SEO specialists, product managers, and others also participate? Who will be responsible for ensuring it’s running the way you want and improving user satisfaction?

In-house configuration of open source software typically requires  search engineering, which is difficult even for experienced engineers. For service based delivery the work of maintenance and optimization can potentially fall upon employees without technical skills.

Even after site search is running and optimized, you can’t ignore it. It requires continual optimization to meet the demands of customers on your site. A best practice is to determine your long-term needs and find a solution that will work for your team.

Security

As mentioned earlier, site search security is important if your index contains sensitive data or you’re limiting access to your content to select users. This is a point that goes beyond the question of how to find the right search solution for your team, but it’s important to identify how important it is for your organization before selecting a search platform.

Ready for a new search engine?

Whew, you made it all the way through. Congratulations. Was this guide helpful? Please drop us a line to let us know.

If you’re considering a new solution, be sure to try Sajari. We offer a fully-functional 14-day free trial, or we would be delighted to provide a live demo to your team.

‍