Full-text search in Azure Cosmos DB for NoSQL

Azure Cosmos DB for NoSQL now offers a powerful full-text search feature. This is designed to enhance the native search capabilities of your apps without needing an external search service for basic full-text search.

What is full-text search?

Full-text indexing and search includes advanced text processing techniques such as stemming, stop word removal, and tokenization, enabling efficient and effective text searches through a specialized text index. Full-text search also includes full-text scoring with a function that evaluates the relevance of documents to a given search query. BM25, or Best Matching 25, considers factors like term frequency, inverse document frequency, and document length to score and rank documents. This helps ensure that the most relevant documents appear at the top of the search results, improving the accuracy and usefulness of text searches.

Full-text search is ideal for various scenarios, including:

• E-commerce: Quickly find products based on descriptions, reviews, and other text attributes.
• Content management: Efficiently search through articles, blogs, and documents.
• Customer support: Retrieve relevant support tickets, FAQs, and knowledge base articles.
• User content: Analyze and search through user-generated content such as posts and comments.
• RAG for chatbots: Enhance chatbot responses by retrieving relevant information from large text corpora, improving the accuracy and relevance of answers.
• Multi-agent AI apps: Enable multiple AI agents to collaboratively search and analyze vast amounts of text data, providing comprehensive and nuanced insights.

How to use full-text search

1. Configure a container with a full-text policy and full-text index.
2. Insert your data with text properties.
3. Run queries against the data using full-text search system functions.

Configure container policies and indexes for hybrid search

To use full-text search capabilities, you should first define two policies:

• A container-level full-text policy that defines what paths contain text for the new full-text query system functions.
• A full-text index added to the indexing policy that enables efficient search.

While it's possible to run full-text search queries without these policies, they won't utilize the full-text index and can consume higher request units (RUs) and have longer execution times. It's strongly recommended to define full-text container and index policies.

Full-text policy

For every text property you'd like to configure for full-text search, you must declare both the path of the property with text and the language of the text. A simple full-text policy can be:

{
   "defaultLanguage": "en-US",
   "fullTextPaths": [
       {
           "path": "/text",
           "language": "en-US"
       }
   ]
}

Defining multiple text paths is easily done by adding another element to the fullTextPaths array:

{
   "defaultLanguage": "en-US",
   "fullTextPaths": [
       {
           "path": "/text1",
           "language": "en-US"
       },
       {
           "path": "/text2",
           "language": "en-US"
       }
   ]
}

You can also use array wildcard notation to index array paths. For example:

{
    "defaultLanguage": "en-US",
    "fullTextPaths": [
        {
            "path": "/text/[]",
            "language": "en-US"
        },
        {
            "path": "/text/[]/text2",
            "language": "en-US"
        }
    ]
}

Multi-language support (preview)

Multi-language support allows you to index and search text in languages beyond English. It applies language-specific tokenization, stemming, and stopword removal for more accurate search results.

Currently supported languages are:

• en-US (English)
• de-DE (German)
• es-ES (Spanish)
• fr-FR (French)
• it-IT (Italian)
• pt-PT (Portuguese)
• pt-BR (Brazilian Portuguese)

Full-text index

Any full-text search operations should make use of a full-text index. A full-text index can easily be defined in any Azure Cosmos DB for NoSQL index policy per the following example.

{
    "indexingMode": "consistent",
    "automatic": true,
    "includedPaths": [
        {
            "path": "/*"
        }
    ],
    "excludedPaths": [
        {
            "path": "/\"_etag\"/?"
        },
    ],
    "fullTextIndexes": [
        {
            "path": "/text"
        }
    ]
}

As with the full-text policies, full-text indexes can be defined on multiple paths.

{
    "indexingMode": "consistent",
    "automatic": true,
    "includedPaths": [
        {
            "path": "/*"
        }
    ],
    "excludedPaths": [
        {
            "path": "/\"_etag\"/?"
        },
    ],
    "fullTextIndexes": [
        {
            "path": "/text"
        },
        {
            "path": "/text2"
        }
    ]
}

Full-text search queries

Full-text search and scoring operations are performed using the following system functions in the Azure Cosmos DB for NoSQL query language:

• FullTextContains: Returns true if a given string is contained in the specified property of a document. This is useful in a WHERE clause when you want to ensure specific key words are included in the documents returned by your query.
• FullTextContainsAll: Returns true if all of the given strings are contained in the specified property of a document. This is useful in a WHERE clause when you want to ensure that multiple key words are included in the documents returned by your query.
• FullTextContainsAny: Returns true if any of the given strings are contained in the specified property of a document. This is useful in a WHERE clause when you want to ensure that at least one of the key words is included in the documents returned by your query.
• FullTextScore: Returns a score. This can only be used in an ORDER BY RANK clause, where the returned documents are ordered by the rank of the full-text score, with most relevant (highest scoring) documents at the top, and least relevant (lowest scoring) documents at the bottom.

Here are a few examples of each function in use.

FullTextContains

In this example, we want to obtain the first 10 results where the phrase red bicycle is contained in the property c.text.

SELECT TOP 10 *
FROM c
WHERE FullTextContains(c.text, "red bicycle")

FullTextContainsAll

In this example, we want to obtain first 10 results where the keywords red and bicycle are contained in the property c.text, but not necessarily together.

SELECT TOP 10 *
FROM c
WHERE FullTextContainsAll(c.text, "red", "bicycle")

FullTextContainsAny

In this example, we want to obtain the first 10 results where the keywords red and either bicycle or skateboard are contained in the property c.text.

SELECT TOP 10 *
FROM c
WHERE FullTextContains(c.text, "red") AND FullTextContainsAny(c.text, "bicycle", "skateboard")

FullTextScore

In this example, we want to obtain the first 10 results where mountain and bicycle are included, and sorted by order of relevance. That is, documents that have these terms more often should appear higher in the list.

SELECT TOP 10 *
FROM c
ORDER BY RANK FullTextScore(c.text, "bicycle", "mountain")

Fuzzy search

Fuzzy search can improve resilience to typos and text variations. You can specify an allowable distance (number of edits) between the search term and document text, allowing near matches to be considered a hit. The maximum distance that can be specified is 2 (two edits).

The following example query retrieves documents where the text includes words similar to red (within one edit) and bycycle (within two edits):

SELECT TOP 10 *
FROM c
WHERE FullTextContains(c.text, {"term": "red", "distance":1}) AND FullTextContains(c.text, {"term": "bycycle", "distance":2})

Faceting

Faceting allows you to provide aggregated summaries of search results, making it easier for users to filter and explore matching documents by common attributes.

Azure Cosmos DB doesn't provide a dedicated faceting operator. Instead, you can achieve faceting functionality by combining full-text search with aggregation queries such as GROUP BY, COUNT, and COUNTIF.

For example, the following query returns the number of matching documents in each category for documents that contain the term shoes:

SELECT
    c.category AS facetKey,
    COUNT(1) AS facetCount
FROM c
WHERE FullTextContains(c.title, "shoes")
GROUP BY c.category

You can also generate facets from numeric values. For example, the following query returns the number of matching documents grouped by rating:

SELECT
    FLOOR(c.rating) AS facetKey,
    COUNT(1) AS facetCount
FROM c
WHERE FullTextContains(c.title, "shoes")
    OR FullTextContains(c.description, "shoes")
    OR FullTextContains(c.brand, "shoes")
GROUP BY FLOOR(c.rating)

Some facets are based on ranges rather than grouping. For example, a price-range facet can be implemented by using aggregate functions:

SELECT
    COUNTIF(c.price < 25) AS under25,
    COUNTIF(c.price >= 25 AND c.price < 50) AS r25to50,
    COUNTIF(c.price >= 50 AND c.price < 100) AS r50to100,
    COUNTIF(c.price >= 100) AS over100
FROM c
WHERE FullTextContains(c.title, "shoes")

Use these patterns to generate facet summaries alongside full-text search results, so users can refine and explore search results more effectively.

Related content

• FullTextContains system function
• FullTextContainsAll system function
• FullTextContainsAny system function
• FullTextScore system function
• RRF system function
• ORDER BY RANK clause