Package org.apache.lucene.sandbox.codecs.dedup

Class DedupHnswScalarQuantizedVectorsFormat

java.lang.Object

org.apache.lucene.codecs.KnnVectorsFormat

org.apache.lucene.sandbox.codecs.dedup.DedupHnswScalarQuantizedVectorsFormat

All Implemented Interfaces:

NamedSPILoader.NamedSPI

public final class DedupHnswScalarQuantizedVectorsFormat
extends KnnVectorsFormat

An HNSW vector format that de-duplicates vectors, storing each distinct vector once in both raw and scalar quantized form.

Graph construction and search are identical to Lucene99HnswVectorsFormat, with searches scoring against the quantized vectors. A DedupScalarQuantizedVectorsFormat is used for the flat vector storage, which stores each distinct vector exactly once, shared across all documents that reference it. See DedupHnswVectorsFormat for the intended multi-field usage pattern.

If you customize this format, be sure to share the same instance of the underlying DedupScalarQuantizedVectorsFormat to de-duplicate vectors correctly.

WARNING: This API is experimental and might change in incompatible ways in the next release.

Field Summary

Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat

DEFAULT_MAX_DIMENSIONS, EMPTY

Constructor Summary

Constructors

Constructor	Description
DedupHnswScalarQuantizedVectorsFormat()	Constructs a format with UNSIGNED_BYTE encoding and default graph construction parameters
DedupHnswScalarQuantizedVectorsFormat(int maxConn, int beamWidth)	Constructs a format with UNSIGNED_BYTE encoding, using the given graph construction parameters.
DedupHnswScalarQuantizedVectorsFormat(QuantizedByteVectorValues.ScalarEncoding encoding, int maxConn, int beamWidth)	Constructs a format using the given quantization encoding and graph construction parameters.
DedupHnswScalarQuantizedVectorsFormat(QuantizedByteVectorValues.ScalarEncoding encoding, int maxConn, int beamWidth, int numMergeWorkers, ExecutorService mergeExec)	Constructs a format using the given quantization encoding and graph construction parameters.
DedupHnswScalarQuantizedVectorsFormat(QuantizedByteVectorValues.ScalarEncoding encoding, int maxConn, int beamWidth, int numMergeWorkers, ExecutorService mergeExec, int tinySegmentsThreshold)	Constructs a format using the given quantization encoding and graph construction parameters.

Method Summary

Modifier and Type	Method	Description
KnnVectorsReader	fieldsReader(SegmentReadState state)
KnnVectorsWriter	fieldsWriter(SegmentWriteState state)
int	getMaxDimensions(String fieldName)
String	toString()

Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat

availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

DedupHnswScalarQuantizedVectorsFormat

public DedupHnswScalarQuantizedVectorsFormat()

Constructs a format with UNSIGNED_BYTE encoding and default graph construction parameters

DedupHnswScalarQuantizedVectorsFormat

public DedupHnswScalarQuantizedVectorsFormat(int maxConn,
 int beamWidth)

Constructs a format with UNSIGNED_BYTE encoding, using the given graph construction parameters.

Parameters:

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

DedupHnswScalarQuantizedVectorsFormat

public DedupHnswScalarQuantizedVectorsFormat(QuantizedByteVectorValues.ScalarEncoding encoding,
 int maxConn,
 int beamWidth)

Constructs a format using the given quantization encoding and graph construction parameters.

Parameters:

encoding - the quantization encoding used to encode the vectors

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

DedupHnswScalarQuantizedVectorsFormat

public DedupHnswScalarQuantizedVectorsFormat(QuantizedByteVectorValues.ScalarEncoding encoding,
 int maxConn,
 int beamWidth,
 int numMergeWorkers,
 ExecutorService mergeExec)

Constructs a format using the given quantization encoding and graph construction parameters.

Parameters:

encoding - the quantization encoding used to encode the vectors

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

numMergeWorkers - number of workers (threads) that will be used when doing merge. If larger than 1, a non-null ExecutorService must be passed as mergeExec

mergeExec - the ExecutorService that will be used by ALL vector writers that are generated by this format to do the merge. If null, the configured MergeScheduler.getIntraMergeExecutor(MergePolicy.OneMerge) is used.

DedupHnswScalarQuantizedVectorsFormat

public DedupHnswScalarQuantizedVectorsFormat(QuantizedByteVectorValues.ScalarEncoding encoding,
 int maxConn,
 int beamWidth,
 int numMergeWorkers,
 ExecutorService mergeExec,
 int tinySegmentsThreshold)

Constructs a format using the given quantization encoding and graph construction parameters.

Parameters:

encoding - the quantization encoding used to encode the vectors

maxConn - the maximum number of connections to a node in the HNSW graph

beamWidth - the size of the queue maintained during graph construction.

numMergeWorkers - number of workers (threads) that will be used when doing merge. If larger than 1, a non-null ExecutorService must be passed as mergeExec

mergeExec - the ExecutorService that will be used by ALL vector writers that are generated by this format to do the merge. If null, the configured MergeScheduler.getIntraMergeExecutor(MergePolicy.OneMerge) is used.

tinySegmentsThreshold - the expected number of vector operations to return k nearest neighbors of the current graph size

Method Details

fieldsWriter

public KnnVectorsWriter fieldsWriter(SegmentWriteState state)
                              throws IOException

Specified by:

fieldsWriter in class KnnVectorsFormat

Throws:

IOException

fieldsReader

public KnnVectorsReader fieldsReader(SegmentReadState state)
                              throws IOException

Specified by:

fieldsReader in class KnnVectorsFormat

Throws:

IOException

getMaxDimensions

public int getMaxDimensions(String fieldName)

Specified by:

getMaxDimensions in class KnnVectorsFormat

toString

public String toString()

Overrides:

toString in class Object