-
Notifications
You must be signed in to change notification settings - Fork 3.4k
The index factory
The index_factory
function interprets a string to produce a composite Faiss index. The string is a comma-separated list of components.
It is intended to facilitate the construction of index structures, especially if they are nested.
The index_factory
argument typically includes a preprocessing component, and inverted file and an encoding component.
This page summarizes the index_factory
components and arguments.
The numbers we indicate are examples, and d is the input dimension.
String | Class name | Comments |
---|---|---|
IDMap | IndexIDMap |
Used to enable add_with_ids on indexes that do not support it, mainly the flat indexes. |
These strings map to VectorTransform
objects that can be applied on vectors before they are indexed
String | Class name | Output dimension | Comments |
---|---|---|---|
PCA64, PCAR64, PCAW64, PCAWR64 | PCAMatrix |
64 | Applies a PCA transform to reduce the number of dimensions. W = follow with whitening, R = follow with random rotation. The PCAR is especially useful as pre-processing with a Flat or scalar quantizer. |
OPQ16, OPQ16_64 | OPQMatrix |
d, 64 | Rotates the vectors so that they can be encoded efficiently by a PQ to 16 subvectors. 64 is the output dimension, because it is often beneficial to also reduce the number of dimensions. If the output dimensions not specified, it is the same as the input dimension. |
RR64 | RandomRotation |
64 | Random rotation on the input data. The dimension can increase or decrease wrt. the input |
L2norm | NormalizationTransform |
d | L2-normalizes the input vectors |
ITQ256, ITQ | ITQMatrix |
256, d | Applies an ITQ transformation to the input vectors, see "Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval" by Gong et al.. This is useful when the vectors are encoded with LSH. |
Pad128 | RemapDimensionsTransform |
128 | Pad the input vectors with 0s to 128 dim |
Inverted files all inherit from IndexIVF
.
The non-exhaustive component specifies what the coarse quantizer (first parameter of the constructor) should be.
String | Quantizer class | Number of centroids | Comments |
---|---|---|---|
IVF4096 |
IndexFlatL2 or IndexFlatIP
|
4096 | Constructs one of the IndexIVF variants, with a flat quantizer. |
IMI2x9 | MultiIndexQuantizer |
2^(2 * 9) = 262144 | Constructs an IVF with many more centroids, possibly more balanced. |
IVF65536_HNSW32 | IndexHNSWFlat |
65536 | The quantizer is trained as a flat index but indexed with a HNSW. This makes quantization a lot faster |
HNSW indexes inherit from IndexHNSW
.
The IndexHNSW
relies on a flat storage that stores the actual vectors.
The HNSW32
String | Storage class | Comment |
---|---|---|
HNSW32 | IndexFlatL2 |
32 is the number of links in the upper levels of the HNSW, the lowest level that uses most memory has 32 * 2 links, or 32 * 2 * 4 = 256 bytes per vector. This applies to other HSNW variants. |
HNSW32_SQ8 | IndexScalarQuantizer |
SQ8 scalar quantizer |
HNSW32_PQ12 | IndexPQ |
PQ12x8 index |
HNSW32_16384+PQ12 | Index2Layer |
1st layer is a flat index, the PQ encodes the residual of the quantizer |
HNSW32_2x10+PQ12 | Index2Layer |
1st layer is an IMI index, the PQ encodes the residual of the quantizer |
String | Class name (Flat/IVF) | code size (bytes) | Comments |
---|---|---|---|
Flat |
IndexFlat , IndexIVFFlat
|
4 * d | The vectors are stored as is, without any encoding |
PQ16, PQ16x12 |
IndexPQ , IndexIVFPQ
|
16, ceil(16 * 12 / 8) | Uses Product Quantization codes with 16 codes of 12 bits each. When the number of bits is omitted, it is set to 8 (IndexIVFPQ supports only 8-bit encodings. With suffix "np" does not train the Polysemous permutation, which can be slow. |
SQ4, SQ8, SQ6, SQfp16 |
IndexScalarQuantizer , IndexIVF ScalarQuantizer
|
4*d/8 , d, 6*d/8 , d*2
|
Scalar quantizer encoding |
Residual128, Residual3x10 | Index2Layer |
ceil(log2(128) / 8) , ceil(3*10 / 8)
|
Residual encoding. Quantizes the vectors into 128 centroids or 3x10 MI centroids. Should be followed by PQ or SQ to actually encode the residual. Only for use as a codec. |
ZnLattice3x10_6 | IndexLattice |
ceil((3*log2(C(d/3, 10)) + 6) / 8) |
Lattice codec. The vector is first split into 3 segments, then each segment is encoded as its norm (6 bits) and as a direction on the unit sphere in dimension d/3, quantized by the Zn lattice of radius^2 = 10. C(dim, r2) is the number of points on the sphere of radius sqrt(r2) that have integer coordinates (see here for more details). |
LSH, LSHrt, LSHr, LSHt | IndexLSH |
ceil(d / 8) | Binarizes the vectors by thresholding them. At search time, the query vectors are also binarized (symmetric search). Suffix r = rotate vectors prior to binarization, t = train thresholds to balance 0s and 1s. Useful in combination with ITQ. |
Faiss building blocks: clustering, PCA, quantization
Index IO, cloning and hyper parameter tuning
Threads and asynchronous calls
Inverted list objects and scanners
Indexes that do not fit in RAM
Brute force search without an index
Fast accumulation of PQ and AQ codes (FastScan)
Setting search parameters for one query
Binary hashing index benchmark