Search Playbook

How to build search into a product

Image by Solid Digital

Overview

Search analytics is the practice of analyzing search data to gain insights into user behavior and improve search functionality. It is important because it helps organizations understand user intent, optimize search results, and enhance the overall user experience. I learned about search through building search for the company I founded Netscale, as well as through my Product role on the Search & Discovery role at Roblox.

Product Journey

1. After # documents grows into the hundreds of documents, adding search to the product is critical

2. Quick way to evaluate search is through evaluating a random sample of ~200 queries. You want to evaluate:

   1. Query Type Breakdown

      1. Broad: looking for a category, genre of titles (action movies)

      2. Content: searching through broad meta-descriptions (movies with a dog that the family will love)

      3. Navigational: looking for a very specific title (Die Hard)

   2. Quality of Results Breakdown

      1. Very useful (relevant result in top 6)

      2. Useful (relevant result in top 16)

      3. Not useful (outside of top 16 results)

3. Improve search by building out the 3 components of search

   1. Search Expression: Correcting user inputs through synonyms and spell checking

   2. Result Generation: Candidate generation and objective ranking function

   3. Result Presentation: Showing results in meaningful manner to user

4. Improve search through leveraging historical user search data and analyzing what rank the result clicked on is at → move results clicked on for a query up

5. As you get more confident that the right result in the top rankings for navigational queries, you can start to show recommendations, discovery algorithms to guide the user to new content

Metrics

• Query distribution: Which queries are most common, can breakdown by user type

• Search volume: The total number of searches performed within a specific time frame.

• Click-through rate (CTR): The percentage of users who click on a search result after performing a search.

• Conversion rate: The percentage of users who complete a desired action (e.g., making a purchase) after performing a search.

• Zero-result rate: The percentage of searches that return no results.

• Average search session duration: The average time users spend on the search results page.

• Popular search terms: The most frequently searched keywords or phrases.

• Top search queries with low CTR: Identifying queries that generate a high number of impressions but have a low click-through rate.