Package org.apache.lucene.sandbox.codecs.dedup

Class DedupHnswVectorsFormat

java.lang.Object

org.apache.lucene.codecs.KnnVectorsFormat

org.apache.lucene.sandbox.codecs.dedup.DedupHnswVectorsFormat

All Implemented Interfaces:

NamedSPILoader.NamedSPI

public final class DedupHnswVectorsFormat
extends KnnVectorsFormat

An HNSW vector format that de-duplicates raw vectors.

Graph construction and search are identical to Lucene99HnswVectorsFormat. A DedupFlatVectorsFormat is used for the flat vector storage, which stores each distinct vector exactly once, shared across all documents that reference it.

This format is suitable for high-performance filtered vector search when filter information is available at indexing time. In addition to the primary vector field, the user creates separate fields for each filter value (e.g. product categories in an e-commerce search engine) to build dedicated HNSW graphs that share the same raw vector storage. The responsibility of searching the right field at query-time lies on the user.

This scheme allows for more efficient search than query-time pre-filtering (i.e. AcceptDocs derived from a Query filter) at the expense of slower indexing and larger indexes due to additional HNSW graphs.

If you customize this format, be sure to share the same instance of the underlying DedupFlatVectorsFormat to de-duplicate raw vectors correctly.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Field Summary

Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat

DEFAULT_MAX_DIMENSIONS, EMPTY

Constructor Summary

Constructors

Constructor	Description
DedupHnswVectorsFormat()	Constructs a format using default graph construction parameters
DedupHnswVectorsFormat(int maxConn, int beamWidth)	Constructs a format using the given graph construction parameters.
DedupHnswVectorsFormat(int maxConn, int beamWidth, int tinySegmentsThreshold)	Constructs a format using the given graph construction parameters.
DedupHnswVectorsFormat(int maxConn, int beamWidth, int numMergeWorkers, ExecutorService mergeExec)	Constructs a format using the given graph construction parameters.
DedupHnswVectorsFormat(int maxConn, int beamWidth, int numMergeWorkers, ExecutorService mergeExec, int tinySegmentsThreshold)	Constructs a format using the given graph construction parameters.

Method Summary

Modifier and Type	Method	Description
KnnVectorsReader	fieldsReader(SegmentReadState state)
KnnVectorsWriter	fieldsWriter(SegmentWriteState state)
int	getMaxDimensions(String fieldName)
String	toString()

Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat

availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

DedupHnswVectorsFormat

public DedupHnswVectorsFormat()

Constructs a format using default graph construction parameters

DedupHnswVectorsFormat

public DedupHnswVectorsFormat(int maxConn,
 int beamWidth)

Constructs a format using the given graph construction parameters.

Parameters:

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

DedupHnswVectorsFormat

public DedupHnswVectorsFormat(int maxConn,
 int beamWidth,
 int tinySegmentsThreshold)

Constructs a format using the given graph construction parameters.

Parameters:

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

tinySegmentsThreshold - the expected number of vector operations to return k nearest neighbors of the current graph size

DedupHnswVectorsFormat

public DedupHnswVectorsFormat(int maxConn,
 int beamWidth,
 int numMergeWorkers,
 ExecutorService mergeExec)

Constructs a format using the given graph construction parameters.

Parameters:

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

numMergeWorkers - number of workers (threads) that will be used when doing merge. If larger than 1, a non-null ExecutorService must be passed as mergeExec

mergeExec - the ExecutorService that will be used by ALL vector writers that are generated by this format to do the merge. If null, the configured MergeScheduler.getIntraMergeExecutor(MergePolicy.OneMerge) is used.

DedupHnswVectorsFormat

public DedupHnswVectorsFormat(int maxConn,
 int beamWidth,
 int numMergeWorkers,
 ExecutorService mergeExec,
 int tinySegmentsThreshold)

Constructs a format using the given graph construction parameters.

Parameters:

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

numMergeWorkers - number of workers (threads) that will be used when doing merge. If larger than 1, a non-null ExecutorService must be passed as mergeExec

mergeExec - the ExecutorService that will be used by ALL vector writers that are generated by this format to do the merge. If null, the configured MergeScheduler.getIntraMergeExecutor(MergePolicy.OneMerge) is used.

tinySegmentsThreshold - the expected number of vector operations to return k nearest neighbors of the current graph size

Method Details

fieldsWriter

public KnnVectorsWriter fieldsWriter(SegmentWriteState state)
                              throws IOException

Specified by:

fieldsWriter in class KnnVectorsFormat

Throws:

IOException

fieldsReader

public KnnVectorsReader fieldsReader(SegmentReadState state)
                              throws IOException

Specified by:

fieldsReader in class KnnVectorsFormat

Throws:

IOException

getMaxDimensions

public int getMaxDimensions(String fieldName)

Specified by:

getMaxDimensions in class KnnVectorsFormat

toString

public String toString()

Overrides:

toString in class Object