MongoDB - Atlas Vector Search

In this article

Overview

The MongoDB - Atlas Vector Search Snap enables you to efficiently perform advanced vector-based queries. This Snap is useful for Similarity searches, Approximate Nearest Neighbor (ANN) queries, and Range queries on vector data stored in MongoDB Atlas. Learn more about vector search queries.

You can run MongoDB Atlas Vector Search Snap in Groundplex or Cloudplex via the MongoDB Atlas service.

mongodb-atlas-vector-search-overview3.png

Snap Type

The MongoDB - Atlas Vector Search Snap is a Read-type Snap.

Prerequisites

Support for Ultra Pipelines

Works in Ultra Pipelines

Limitations

The listSearchIndexes command can only be run on a deployment hosted on MongoDB Atlas, and requires an Atlas cluster tier of at least M10. Hence, for the suggestions list to populate in the Search index field, you must deploy the MongoDB Atlas cluster with at least an M10 tier. However, you can add the index manually, even if the Snap displays an error when populating the suggestions.

Known Issues

None.

Snap Views

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Type

Format

Number of Views

Examples of Upstream and Downstream Snaps

Description

Input 

Document

 

  • Min: 1

  • Max: 1

  • Join

  • JSON Generator

Requires the MongoDB aggregate command to perform vector search.

Output

Document

 

  • Min: 1

  • Max: 1

  • Mapper

  • Copy

Retrieves query results from the collection as specified in the settings.

Error

Error handling is a generic way to handle errors without losing data or failing the Snap execution. You can handle the errors that the Snap might encounter when running the pipeline by choosing one of the following options from the When errors occur list under the Views tab:

  • Stop Pipeline Execution: Stops the current pipeline execution if the Snap encounters an error.

  • Discard Error Data and Continue: Ignores the error, discards that record, and continues with the remaining records.

  • Route Error Data to Error View: Routes the error data to an error view without stopping the Snap execution.

Learn more about Error handling in Pipelines.

Snap Settings

  • Asterisk ( * ): Indicates a mandatory field.

  • Suggestion icon (): Indicates a list that is dynamically populated based on the configuration.

  • Expression icon ( ): Indicates the value is an expression (if enabled) or a static value (if disabled). Learn more about Using Expressions in SnapLogic.

  • Add icon ( ): Indicates that you can add fields in the field set.

  • Remove icon ( ): Indicates that you can remove fields from the field set.

  • Upload icon ( ): Indicates that you can upload files.

Field Name

Field Type

Description

Field Name

Field Type

Description

Label*

 

Default ValueMongoDB - Atlas Vector Search
ExampleAtlas Vector Search

String

Specify the name for the Snap. You can modify this to be more specific, especially if you have more than one of the same Snap in your pipeline.

 

Database name

 

Default Value: N/A
Example: chunking_strategy

String/Expression/Suggestion

Specify the database where the query is executed. If you do not specify, the database configured in the account is used.

Collection name*

 

Default Value: N/A
Example: chunk_metadata

String/Expression/Suggestion

Specify or select a MongoDB collection name to find indexes for vector search.

Search index*

 

Default Value: N/A
Example: vector_index

String/Expression/Suggestion

Specify the name of the vector search index to query. Alternatively, click the Suggestions icon to select the vector search index.

For search index suggestions to work, you must deploy the MongoDB Atlas cluster with at least an M10 tier. Otherwise, you can add the index manually, even if the Snap displays an error when you click the Suggestion icon.

Vector field*

 

Default Value: N/A
Example: embedding

String/Expression/Suggestion

Specify the name of the vector field that you want to search.

Number of candidates*

 

Default Value100
Example: 50

Integer/Expression

Specify the number of candidates for vector search.

Maximum value: 10000
Minimum value: 0

Limit*

 

Default Value4
Example: 10

Integer/Expression

Specify the number of results to return for each query.

Maximum value: 10000
Minimum value: 0

Advanced settings

Configure the advanced settings as required.

Batch size*

 

Default Value0
Example10

Integer

Specify the number of documents to return per batch.

Timezone hours offset

 

Default Value0
Example12

Integer

Specify the time zone hour offset to apply to all DateTime fields.

Timezone minutes offset

 

Default Value0
Example1

Integer

Specify the time zone minutes offset to apply to all DateTime fields.

Number of retries

Default Value0
Example: 4

Integer/Expression

Specify the maximum number of attempts to be made to receive a response. The request is terminated if the attempts do not result in a response.

Retry interval (seconds)

Default Value: 1
Example: 5

Integer/Expression

Specify the time interval between two retry requests.

Snap Execution

 

Default Value
Example: Validate & Execute

Dropdown list

Select one of the following three modes in which the Snap executes:

  • Validate & Execute: Performs limited execution of the Snap, and generates a data preview during pipeline validation. Subsequently, performs full execution of the Snap (unlimited records) during pipeline runtime.

  • Execute only: Performs full execution of the Snap during pipeline execution without generating preview data.

  • Disabled: Disables the Snap and all Snaps that are downstream from it.

Troubleshooting

Error

Reason

Resolution

Error

Reason

Resolution

The vector search operation failed.

The specified vector might be invalid.

Specify a valid vector and retry.

Unable to process the input document.

The input document might be invalid.

Specify valid input and retry.

The collection provided cannot be found.

The specified collection is not available.

Specify a valid collection and retry.

The index is not specified.

The index field is mandatory.

Specify a valid index and retry.

The input document does not have a 'vector' specified in the structure.

The input document must have a valid vector.

Add the vector to query the input document and retry.

  • Unable to retrieve index fields, OR

  • No indexes were found querying database <name>, OR

  • Command failed with error 8000 (AtlasError)

Unrecognized pipeline stage <name-of-the-pipeline>.

Ensure the cluster runs on MongoDB Atlas and has a cluster tier of at least M10. Additionally, confirm that the user has the required permissions to access the search index.

Examples

Vector search on a specific field

Th following example pipeline demonstrates how to use MongoDB - Atlas Vector Search Snap to perform an ANN (approximate nearest neighbor) search on a vector in the specified field.

mongodb-atlas-vector-search-pipeline-overview.png

Download this pipeline.

Step 1: Configure the Edit JSON field in the JSON Generator Snap with the prompt List movies.

Step 2: Configure the Azure OpenAI Embedder Snap as follows to create a vector embedding for the list of movies.

Azure OpenAI Embedder configuration

Azure OpenAI Embedder output

Azure OpenAI Embedder configuration

Azure OpenAI Embedder output

 

Step 3: Configure the Mapper Snap to pass the vector embedding to the MongoDB - Atlas Vector Search Snap.

Step 4: Configure the MongoDB - Atlas Vector Search Snap to run a search query on 10,000 nearest neighbor records for a single-matched record with a similar plot_embedding vector field.

On validation, one record is populated with a similar plot-embedding vector field.

Downloads

 

  File Modified

File example-mongodb-avs.slp

May 02, 2024 by Lakshmi Manda

Related links:

https://www.mongodb.com/docs/atlas/

https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/

https://www.mongodb.com/docs/atlas/atlas-vector-search/tutorials/vector-search-quick-start/