Skip to content

Commit 8682e52

Browse files
docs/search: explain indexed vs unindexed trigram difference (#2084)
Zoekt skips the content of files with more than 20,000 unique trigrams (mostly minified JS, source maps and lockfiles), but unindexed search (searcher) has no such check, so the same query can return more results with `index:no` than indexed. Rather than adding a trigram check to searcher, we agreed to document the difference and the workarounds: `index:no` or `search.largeFiles`. Discussion: https://sourcegraph.slack.com/archives/C022SPMNR0W/p1791537536129499 Co-authored-by: Amp <amp@ampcode.com>
1 parent 86f3ac0 commit 8682e52

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

‎docs/admin/search.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@ To view which files are skipped during indexing, visit the repository settings p
7878
7979
To force the indexer to include specific files (like `yarn.lock` or other large text files) that are otherwise skipped, add their file path or a glob pattern to the [search.largeFiles](/admin/search#maximum-file-size) setting in your advanced configuration and reindex the repository. Note that files must still be valid UTF-8 to be indexed, even if added to `search.largeFiles`.
8080
81+
The unique trigram limit applies only to indexed search. Unindexed search, which runs for unindexed branches and revisions and for queries with `index:no`, still searches the content of these files, so the same query can return more results unindexed than indexed. Files skipped this way are mostly minified JavaScript, source maps, and lockfiles. Their paths are still indexed, so they show up in `type:path` searches. To find content matches in them, add `index:no` to the query, or add them to `search.largeFiles` as described above.
82+
8183
For large deployments we recommend horizontally scaling indexed search. You can do this by adjusting the [number of replicas](/self-hosted/deploy/kubernetes/configure). Sourcegraph shards repository indexes across replicas. When the replica count changes Sourcegraph will slowly rebalance indexes to ensure availability of existing indexes.
8284
8385
The resource requirements for indexed search vary considerably based on the text contents of your repositories, but a good estimate is that the node should have enough memory to hold the entire text contents of the default branch of each repository.

0 commit comments

Comments
 (0)