A Memo on Improving Docusaurus Algolia Search Accuracy with a Single-Index Setup

If this site helped you, please support us with 🌟 ☕️

The HKdocs site search suffered an accuracy decline where "new articles wouldn't show up, yet irrelevant articles kept surfacing." The cause was not a lack of Algolia tuning, but a structural problem: the index the frontend queried and the index the crawler updated were mismatched.

This article records the procedure, from isolating that cause to restoring accuracy by consolidating into a single-index setup. Splitting into per-language indexes is unnecessary even on a multi-language site, and understanding the correct behavior of contextualSearch is the starting point for the fix.

Prerequisite: The correct behavior of contextualSearch

Docusaurus's contextualSearch: true does not change indexName. It queries a single index and narrows results by language at query time with facetFilters (such as language:<locale>). Therefore, even on a multi-language site, splitting into per-language indexes is unnecessary; language filtering is achieved with a single index + facets. Per-language tuning can also be completed within a single index via language-array specifications such as ignorePlurals: ["en"].

Diagnosis: Isolating the Accuracy Decline

When search accuracy declines, before diving into fine-grained tuning, the quickest path is to first suspect whether the frontend's query target matches the crawler's update target. Specifically, check these three points.

algolia.indexName in docusaurus.config.ts (what the frontend queries).
actions[].indexName in the Crawler (what the crawler updates).
The last-updated timestamp and record count of each index (Algolia dashboard).

If the indexName values don't match, or if updates to the frontend's query target have stopped, the frontend keeps querying a stale index. The symptom "new articles don't match / old irrelevant articles surface" is exactly this sign.

Orphaned indexes are hard to notice

If you split into per-language indexes while leaving indexName in docusaurus.config.ts set to the pre-split name, the crawler keeps updating the post-split indexes while the frontend keeps querying the old index that no one updates. Because the crawler itself runs normally and the record counts on the dashboard look healthy, the single fact that "the frontend isn't referencing it" becomes a black box and surfaces only later as a decline in accuracy.

The Approach: Consolidating into a Single Index

The approach is simple. Consolidate the crawler into a single index so that the frontend's query target matches the update target. Since language filtering is handled by contextualSearch: true (the language facet), no change to docusaurus.config.ts is needed. At the same time as this consolidation, I also performed noise removal and language tuning together.

Here is the overall picture of the changes.

Item	Recommended setting	Purpose
indexName	Consolidate into a single name	Match the queried and updated indexes to restore freshness
exclusionPatterns	Exclude tags / authors / category / page / search, etc.	Remove noise from thin pages
content	Add `article pre` to `article p, li, td`	Include code blocks in search
ignorePlurals	`["en"]`	Treat plurals as identical for English only, in a single index
indexLanguages / queryLanguages	`["ja", "en"]`	Dictionary segmentation for Japanese (improves compound-word accuracy)
minWordSizefor1Typo / 2	`4` / `8`	Suppress excessive typo matches in Japanese

Implementation Files and Steps in Detail

Below are the highlights of the settings and steps I actually applied.

1. Crawler Configuration (Consolidating into a Single Index)

Consolidate actions into a single index, and exclude noise pages with exclusionPatterns. Use the language check within recordExtractor only to switch the default value of lvl0; do not split the index itself.

crawler-config.js
new Crawler({
  appId: "YOUR_APP_ID",
  apiKey: "YOUR_CRAWLER_API_KEY", // The admin API key for the crawler
  maxUrls: 5000,
  indexPrefix: "",
  rateLimit: 8,
  renderJavaScript: true,
  startUrls: [
    "https://your-docusaurus-site.com/",
    "https://your-docusaurus-site.com/en/",
  ],
  discoveryPatterns: ["https://your-docusaurus-site.com/**"],
  sitemaps: ["https://your-docusaurus-site.com/sitemap.xml"],

  // --- Exclude noise pages ---
  exclusionPatterns: [
    "https://your-docusaurus-site.com/**/tags/**",
    "https://your-docusaurus-site.com/**/authors/**",
    "https://your-docusaurus-site.com/**/page/**",
    "https://your-docusaurus-site.com/docs/category/**",
    "https://your-docusaurus-site.com/en/docs/category/**",
    "https://your-docusaurus-site.com/search/**",
    "https://your-docusaurus-site.com/en/search/**",
  ],

  schedule: "at 21:10 on Tuesday",
  maxDepth: 10,

  // --- Consolidate into a single index ---
  actions: [
    {
      indexName: "yoursite", // Consolidated into one
      pathsToMatch: ["https://your-docusaurus-site.com/**"],
      recordExtractor: ({ url, $, helpers }) => {
        const isEn =
          url.pathname === "/en" || url.pathname.startsWith("/en/");
        const lvl0 =
          $(
            ".menu__link.menu__link--sublist.menu__link--active, .navbar__item.navbar__link--active",
          )
            .last()
            .text() || (isEn ? "Documentation" : "Documentation");

        return helpers.docsearch({
          recordProps: {
            lvl0: { selectors: "", defaultValue: lvl0 },
            lvl1: ["header h1", "article h1"],
            lvl2: "article h2",
            lvl3: "article h3",
            lvl4: "article h4",
            lvl5: "article h5, article td:first-child",
            lvl6: "article h6",
            // Include code blocks (article pre) in the search target
            content:
              "article p, article li, article td:last-child, article pre",
          },
          aggregateContent: true,
          recordVersion: "v3",
        });
      },
    },
  ],
  ignoreCanonicalTo: true,
  safetyChecks: { beforeIndexPublishing: { maxLostRecordsPercentage: 10 } },
});

Don't create the language field yourself

The language field automatically added by helpers.docsearch is enough for the contextualSearch facet filtering to work. There is no need to create a lang field yourself.

2. Index Settings (Language Tuning)

Language tuning is done within a single index by specifying language arrays. Specifying indexLanguages: ["ja", "en"] enables dictionary segmentation for Japanese, improving compound-word search accuracy.

index settings (injected via setSettings)
{
  ignorePlurals: ["en"],           // Treat plurals as identical for English only
  indexLanguages: ["ja", "en"],    // Enable dictionary segmentation for Japanese
  queryLanguages: ["ja", "en"],
  minWordSizefor1Typo: 4,          // Suppress excessive typo matches in Japanese
  minWordSizefor2Typos: 8,
  attributesForFaceting: ["type", "language", "docusaurus_tag"],
  searchableAttributes: [
    "unordered(hierarchy.lvl0)",
    "unordered(hierarchy.lvl1)",
    "unordered(hierarchy.lvl2)",
    "unordered(hierarchy.lvl3)",
    "unordered(hierarchy.lvl4)",
    "unordered(hierarchy.lvl5)",
    "unordered(hierarchy.lvl6)",
    "content",
  ],
  customRanking: [
    "desc(weight.pageRank)",
    "desc(weight.level)",
    "asc(weight.position)",
  ],
  removeWordsIfNoResults: "allOptional",
}

initialIndexSettings is not automatically applied to an existing index

initialIndexSettings is applied only when an index is newly created. To reflect the settings on an existing index, you must inject them directly with setSettings (the REST PUT .../settings).

3. Frontend Configuration (No Changes Needed)

As long as contextualSearch: true is enabled, indexName can remain a single name. Language filtering is performed automatically by the facet.

docusaurus.config.ts
themeConfig: {
  algolia: {
    appId: 'YOUR_APP_ID',
    apiKey: 'YOUR_SEARCH_ONLY_API_KEY', // Search-only key
    indexName: 'yoursite',              // A single name is fine
    contextualSearch: true,             // Narrow by language with the language facet
  },
},

4. Applying Changes via the Crawler REST API

The "Review and publish" in the admin UI may be unusable when Domain Verification is disabled (the DocSearch Status field is blank). In that case, update the production config directly via the Crawler REST API. This path bypasses the UI gate, and the existing scheduled crawls run on this API configuration as well.

# Auth: --user "CRAWLER_USER_ID:CRAWLER_API_KEY"

# Get config
GET   https://crawler.algolia.com/api/1/crawlers/{crawlerId}?withConfig=true

# Update config (consolidate actions into a single index + add exclusionPatterns)
PATCH https://crawler.algolia.com/api/1/crawlers/{crawlerId}/config

# Index settings (inject language tuning directly)
PUT   https://{APP_ID}.algolia.net/1/indexes/yoursite/settings
      # Header: X-Algolia-API-Key: <admin API key for the index>

# Re-crawl (run before and after applying settings)
POST  https://crawler.algolia.com/api/1/crawlers/{crawlerId}/reindex

Gotchas when applying via the API

recordExtractor must be sent in the {"__type":"function","source":"..."} format. Sending it as a plain string results in a validation_error. Also, after changing settings, always run reindex, then verify Before / After by running real queries with the public search key.

Operational Notes

To operate safely, I follow these rules.

First, I never publish from the UI. The UI editor may still contain old code from the split era, and publishing from there rolls back the improvements. All configuration changes are made via the API.

Next, since the Crawler API key and the admin API key for the index used for the work have admin privileges, I rotate them after the work. Finally, after any change, I run representative queries with the public search key to confirm the intended articles rank at the top and the exclusion patterns return 0 results.

Summary

When facing a search accuracy decline, it proved effective to first confirm whether the frontend's reference matches the crawler's update target, rather than jumping straight into fine-grained tuning. Because contextualSearch does not change indexName, even on a multi-language site, language filtering is achieved with a single index + the language facet, and splitting into per-language indexes is generally unnecessary. Per-language tuning can also be fully achieved within a single index via language-array specifications such as ignorePlurals: ["en"] and indexLanguages: ["ja", "en"].

Diagnosis: Isolating the Accuracy Decline​

The Approach: Consolidating into a Single Index​

Implementation Files and Steps in Detail​

1. Crawler Configuration (Consolidating into a Single Index)​

2. Index Settings (Language Tuning)​

3. Frontend Configuration (No Changes Needed)​

4. Applying Changes via the Crawler REST API​

Operational Notes​

Summary​