-
Notifications
You must be signed in to change notification settings - Fork 3.4k
The index factory
The index_factory
function interprets a string to produce a composite Faiss index. The string is a comma-separated list of components.
It is intended to facilitate the construction of index structures, especially if they are nested.
The index_factory
argument typically includes a preprocessing component, and inverted file and an encoding component.
This page summarizes the index_factory
components and arguments.
The numbers we indicate are examples, and d is the input dimension.
String | Class name | Comments |
---|---|---|
IDMap | IndexIDMap |
Used to enable add_with_ids on indexes that do not support it, mainly the flat indexes. |
These strings map to VectorTransform
objects that can be applied on vectors before they are indexed
String | Class name | Output dimension | Comments |
---|---|---|---|
PCA64, PCAR64, PCAW64, PCAWR64 | PCAMatrix |
64 | Applies a PCA transform to reduce the number of dimensions. W = follow with whitening, R = follow with random rotation. The PCAR is especially useful as pre-processing with a Flat or scalar quantizer. |
OPQ16, OPQ16_64 | OPQMatrix |
d, 64 | Rotates the vectors so that they can be encoded efficiently by a PQ to 16 subvectors. 64 is the output dimension, because it is often beneficial to also reduce the number of dimensions. If the output dimensions not specified, it is the same as the input dimension. |
RR64 | RandomRotation |
64 | Random rotation on the input data. The dimension can increase or decrease wrt. the input |
L2norm | NormalizationTransform |
d | L2-normalizes the input vectors |
ITQ256, ITQ | ITQMatrix |
256, d | Applies an ITQ transformation to the input vectors, see "Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval" by Gong et al.. This is useful when the vectors are encoded with LSH. |
Pad128 | RemapDimensionsTransform |
128 | Pad the input vectors with 0s to 128 dim |
For the inverted file, it supports:
-
the flat index "IVF4096" means building an inverted file of size 4096 with a flat coarse quantizer
IndexFlatL2
. -
the inverted multi-index "IMI2x8" means building an inverted file of size 2^(2*8) with an inverted multi-index quantizer using 2x8 bits (uses a
MultiIndexQuantizer
) -
if you do not plan to use an inverted file but need
add_with_ids
, you can set "IDMap" to makeIndexIDMap
that encapsulates the index.
String | Class name (Flat/IVF) | code size (bytes) | Comments |
---|---|---|---|
Flat |
IndexFlat , IndexIVFFlat
|
4 * d | The vectors are stored as is, without any encoding |
PQ16, PQ16x12 |
IndexPQ , IndexIVFPQ
|
16, ceil(16 * 12 / 8) | Uses Product Quantization codes with 16 codes of 12 bits each. When the number of bits is omitted, it is set to 8 (IndexIVFPQ supports only 8-bit encodings. With suffix "np" does not train the Polysemous permutation, which can be slow. |
SQ4, SQ8, SQ6, SQfp16 |
IndexScalarQuantizer , IndexIVFScalarQuantizer
|
4*d/8 , d, 6*d/8 , d*2
|
Scalar quantizer encoding |
Residual128, Residual3x10 | Index2Layer |
ceil(log2(128) / 8) , ceil(3*10 / 8)
|
Residual encoding. Quantizes the vectors into 128 centroids or 3x10 MI centroids. Should be followed by PQ or SQ to actually encode the residual. Only for use as a codec. |
ZnLattice3x10_6 | IndexLattice |
ceil((3*log2(C(d/3, 10)) + 6) / 8) |
Lattice codec. The vector is first split into 3 segments, then each segment is encoded as its norm (6 bits) and as a direction on the unit sphere in dimension d/3, quantized by the Zn lattice of radius^2 = 10. C(dim, r2) is the number of points on the sphere of radius sqrt(r2) that have integer coordinates (see here for more details). |
LSH, LSHrt, LSHr, LSHt | IndexLSH |
ceil(d / 8) | Binarizes the vectors by thresholding them. At search time, the query vectors are also binarized (symmetric search). Suffix r = rotate vectors prior to binarization, t = train thresholds to balance 0s and 1s. Useful in combination with ITQ. |
Faiss building blocks: clustering, PCA, quantization
Index IO, cloning and hyper parameter tuning
Threads and asynchronous calls
Inverted list objects and scanners
Indexes that do not fit in RAM
Brute force search without an index
Fast accumulation of PQ and AQ codes (FastScan)
Setting search parameters for one query
Binary hashing index benchmark