Atlas Search

Global Search

We needed the ability to search across all our data, or specified subsets (specified entities) of our data. I chose to create a single collection in the database that stored all the search documents. This model allows us to search across all or a subset of entities with absolutely no issues.

The strategy used is to extract all text values and store in a values array in the document stored in the collection configured with search index. For our purpose we also needed a couple of additional array fields for very specific use cases - length, and date. Similarly, we also have an optional location field, that allows proximity searches across relevant entities.

Document Structure

I developed a very simple document structure which represents all our entities. The document has the following fields:

• metadata - A standard structure that captures the created and modified (last modified) dates for an entity, along with the user who last edited the entity.

• usernames - An array of platform username values that represents the list of users who have edited the source document at any point in its history. This allows us to quickly find all entities that were edited at any time by a user. We use this to show the most recently edited entities by the current logged-in user.

• type - An enumeration that identifies the entity that is being indexed. This is the field specified by API clients when they wish to restrict search results to a specified type(s) of entities. This also helps callers to identity the type of entity that is included in the search results.

• values - An array of string values. All string values are extracted (recursively, and as appropriate from referenced entities) from the source entity and stored in the array. Search queries are matched against the values in this array.

• location - An object that stores the geo-coordinates and postal address for an entity. Not all entities have a location property, so location search will naturally be limited to only those entities have a location.

• length - An array of length values. Similar to location, only a subset of entities have length values. Multiple length values can be associated with a single entity, which is why this is an array.

• date - An array of date values. Similar to location and length, only a subset of entities have additional date values to index.

• entity - Un-indexed stored field that stores the entire source document. The entity _id is also used as the _id of the search document. This also has the effect of making it extremely simple to determine which database:collection an _id in our system belongs to.

With this structure, we are able to index all the properties that are of interest to us across all our entities. The stored entity field provides access to the source document direct from the search index, without needing to reach back into the BSON document stored in the search collection, or needing to reach back into the original database:collection.

Mapping

We use a static mapping scheme for the documents in the collection. We need only a subset of the fields in the documents to be indexed. We also store the type and entity fields in the search index, to enable direct use of the results from the search engine, without needing to go back to the database for the original entities.

Note: We use the storedSource option to specify the fields we wish to be stored in the search index. We make use of this by specifying returnStoredSource: true in the $search aggregation query. This avoids expensive fetches of the full document from the indexed collection, especially if there are further stages in the aggregation pipeline.

We specify a $project stage to retrieve the stored fields, as well as the search relevance.

API Model

A simple JSON structure is used as the interface for executing searches against the search index.

• query An array of string values. Each entry in the array is used to compose the search query. If the entry consists of multiple words, the search results will include documents that match any word (OR query). If multiple words are to be treated as a phrase, enclose them in double quotes ("). Each item in the array is treated as a required match for the search (the rule about separate words being treated as an OR still applies), allowing caller to compose an AND query involving multiple words/phrases.

• exclude Word or words that should not match the search results. As with query, individual words are treated as an OR clause, allowing caller to specify multiple words/phrases that must not match the search results.

• range Allows caller to optionally limit search results to only documents that were created or modified within a specified time interval. Range also supports an additional date type which matches against an array of dates extracted from a few entities.

• length Allows caller to optionally limit search results to only documents that have a length value that falls within the specified lower and upper bounds.

• entities Allows caller to optionally restrict matches to only the specified domain object types. If not specified or empty, matches all types of domain objects.

• username Allows caller to optionally limit search results to only documents that were edited by the specified user. This is primarily useful for forensics purposes.