Summary:
**Bugs:**
When following rocksdb_ivf demo to build executable file, its output as:
```bash
faiss/demos/rocksdb_ivf/RocksDBInvertedLists.h:52:35: error: 'faiss::InvertedListsIterator* faiss_rocksdb::RocksDBInvertedLists::get_iterator(size_t) const' marked 'override', but does not override
   52 |     faiss::InvertedListsIterator* get_iterator(size_t list_no) const override;
      |                                   ^~~~~~~~~~~~
make[2]: *** [CMakeFiles/demo_rocksdb_ivf.dir/build.make:90: CMakeFiles/demo_rocksdb_ivf.dir/RocksDBInvertedLists.cpp.o] Error 1
```

**Solution:**
Add relevant variable `void* inverted_list_contex` corresponding `get_iterator`'s base virtual function.

Pull Request resolved: #3326

Reviewed By: mlomeli1, mdouze

Differential Revision: D55629580

Pulled By: algoriddle

fbshipit-source-id: a12fcacb483e0dd576411ad91a3dd1e0de94abec

…Lists (#3327)

Summary:
Pull Request resolved: #3327

**Context**
1. [Issue 2621](#2621) discuss inconsistency between OnDiskInvertedList and InvertedList. OnDiskInvertedList is supposed to handle disk based multiple Index Shards. Thus, we should name it differently when merging invls from index shard.
2. [Issue 2876](#2876) provides usecase of shifting ids when merging invls from different shards.

**In this diff**,
1. To address #1 above, I renamed the merge_from function to merge_from_multiple without touching merge_from base class.
why so? To continue to allow merge invl from one index to ondiskinvl from other index.

2. To address #2 above, I have added support of shift_ids in merge_from_multiple to shift ids from different shards. This can be used when each shard has same set of ids but different data. This is not recommended if id is already unique across shards.

Reviewed By: mdouze

Differential Revision: D55482518

fbshipit-source-id: 95470c7449160488d2b45b024d134cbc037a2083

Summary:
Pull Request resolved: #3338

add reconstruct_n for GPU IVFFlat

Reviewed By: mdouze

Differential Revision: D55577561

fbshipit-source-id: 47f8b939611e2df7dbcd087129538145f627293c

Summary:
Pull Request resolved: #3349

**Context**
[Issue 3128](#3128) is an enhancement request to support remove_ids for IVFPQFastScan.

Existing mechanism use direct map and iterate over items in selector and use scopecodes and scopeIds to replace item to be removed. Given that codes are packed, it is hard to return single code how it is packed in CodePackerPQ4. Thus, we need a custom implementation to removed_ids.

**In this diff**,
1. We have added custom implementation of remove_ids from BlockInvertedLists which unpack code as it iterate and repack in new position. DirectMap use this remove_id function in BlockInvertedLists for type NoMap in DirectMap.

2. Also, we are throwing exception for other map type in DirectMap i.e. HashTable

Reviewed By: mdouze

Differential Revision: D55723390

fbshipit-source-id: 0017b556bd790765251e778ac48ed37ff3a29a45

…3336)

Summary:
Pull Request resolved: #3336

Issues:
#3269
#3024

List of implemented GPU indices: https://github.com/facebookresearch/faiss/wiki/Faiss-on-the-GPU#implemented-indexes

Reviewed By: mdouze

Differential Revision: D55577576

fbshipit-source-id: 49f490cfba6784661e378acf4de3cce4195bb43b

Differential Revision:
D55723390

Original commit changeset: 0017b556bd79

Original Phabricator Diff: D55723390

fbshipit-source-id: 58d61467b30dd11d27398f9f825162f598896845

Summary:
Pull Request resolved: #3354

**Change was previously reverted because of build failure as change D55577576 removed the definition of FAISS_THROW_IF_MSG**

**Context**
[Issue 3128](#3128) is an enhancement request to support remove_ids for IVFPQFastScan.

Existing mechanism use direct map and iterate over items in selector and use scopecodes and scopeIds to replace item to be removed. Given that codes are packed, it is hard to return single code how it is packed in CodePackerPQ4. Thus, we need a custom implementation to removed_ids.

**In this diff**,
1. We have added custom implementation of remove_ids from BlockInvertedLists which unpack code as it iterate and repack in new position. DirectMap use this remove_id function in BlockInvertedLists for type NoMap in DirectMap.

2. Also, we are throwing exception for other map type in DirectMap i.e. HashTable

Reviewed By: ramilbakhshyiev

Differential Revision: D55858959

fbshipit-source-id: c8a0631495380b7dead36720e4507f4d1900d39f

Summary: Pull Request resolved: #3304

Reviewed By: junjieqi

Differential Revision: D55823369

Pulled By: mdouze

fbshipit-source-id: c0e9f4b85d979758f02e9953f3706b63a846bf22

Summary:
Pull Request resolved: #3362

Add test to Alex' PR

Reviewed By: junjieqi

Differential Revision: D56003946

fbshipit-source-id: 5a8a881d450bc97ae0777d73ce0ce8607ec6b686

Summary:
Pull Request resolved: #3363

'sprintf' is deprecated: This function is provided for compatibility reasons only.  Due to security concerns inherent in the design of sprintf(3), it is highly recommended that you use snprintf(3) instead

{F1484071654}

Reviewed By: kuarora

Differential Revision: D56009251

fbshipit-source-id: ec222cf589ff98b016979058d59fc20cccec8f43

Summary:
LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: dmm-fb

Differential Revision: D56065763

fbshipit-source-id: b0541b8a759c4b6ca0e8753fc24b8c227047bd3d

Summary: Previously this code conformed from clang-format 12.

Reviewed By: igorsugak

Differential Revision: D56065247

fbshipit-source-id: f5a985dd8f8b84f2f9e1818b3719b43c5a1b05b3

Summary:
The CMakeLists.txt in faiss/gpu uses the $<LINK_LIBRARY:WHOLE_ARCHIVE expression which requires at least cmake 3.24.

Pull Request resolved: #3305

Reviewed By: mlomeli1

Differential Revision: D56234500

Pulled By: algoriddle

fbshipit-source-id: dfe7df3379c5250dedec7d1988cffa889fc1c393

Summary:
In this commit ab2b7f5, they changed format based on clang-format-18. However, we still use clang-format-11 in our circle ci job which caused the failure. In this PR, we are going to switch to clang-format-18

Pull Request resolved: #3372

Reviewed By: kuarora

Differential Revision: D56280363

Pulled By: junjieqi

fbshipit-source-id: f832ab2112f762e6000b55a155e3e43fe99071d7

Summary:
Pull Request resolved: #3371

This will never happen because N is fixed at compile time and the buffer is large enough. It is misleading to add error handling code for a case that will never happen.

Reviewed By: kuarora

Differential Revision: D56274458

fbshipit-source-id: ca706f1223dbc97e69d5ac9750b277afa4df80a7

Summary:
The current loop goes from 0 to 31.  It has an if statement to do an assignment for j < 16 and a different assignment for j >= 16.  By unrolling the loop to do the j < 16 and the j >= 16 iterations in parallel the if j < 16 is eliminated and the number of loop iterations is reduced in half.

Then unroll the loop for the j < 16 and the j >=16 to a depth of 2.

This change results in approximately a 55% reduction in the execution time for the bench_ivf_fastscan.py workload on Power 10 when compiled with CMAKE_INSTALL_CONFIG_NAME=Release.

The removal of the if (j < 16) statement and the unrolling of the loop removes branch cycle stall and register dependencies on instruction issue. The result is the unrolled code is able issue instructions earlier thus reducing the total number of cycles required to execute the function.

Pull Request resolved: #3364

Reviewed By: kuarora

Differential Revision: D56455690

Pulled By: mdouze

fbshipit-source-id: 490a17a40d9d4439b1a8ea22e991e706d68fb2fa

Summary:
This pull request is for issue #3330. This patch makes sure that packed code arrays are in big endian format. Kindly let us know if we need any changes or if we can have a better approach.

Pull Request resolved: #3345

Reviewed By: junjieqi

Differential Revision: D55957630

Pulled By: mdouze

fbshipit-source-id: f728f9563f6b942af9d8899b54662d7ceb811206

Summary:
Pull Request resolved: #3361

Fix a few issues in the PR.
Normally all tests should pass on a litlle-endian machine

Reviewed By: junjieqi

Differential Revision: D56003181

fbshipit-source-id: 405dec8c71898494f5ddcd2718c35708a1abf9cb

Summary:
Pull Request resolved: #3383

In this diff, I am fixing minor issues in bench_fw where either certain fields are not accessible when index is build from codec. It also requires index to be discovered using codec alias as index factory is not always available.

In subsequent diff internal to meta will have testcase that execute this path.

Reviewed By: algoriddle

Differential Revision: D56444641

fbshipit-source-id: b7af7e7bb47b20bbb5515a66f41dd24f42459d52

Summary: Fixes #3343

Reviewed By: kuarora, junjieqi

Differential Revision: D56526842

fbshipit-source-id: b7c4377495db4e68283cf4ce2b7c8fae008cd404

Summary:
The osx failed

https://app.circleci.com/pipelines/github/facebookresearch/faiss/5698/workflows/4e029c32-8d8b-4db7-99e2-8e802aad6653/jobs/32701

Pull Request resolved: #3357

Reviewed By: kuarora

Differential Revision: D56039739

Pulled By: junjieqi

fbshipit-source-id: dd434a8817148364797eae39c09e0e1e9edbe858

Summary: Remove debugging log lines

Reviewed By: mlomeli1

Differential Revision: D56626636

fbshipit-source-id: 2721b84e4e1359d1372df2b2c95cc668c6a75c3f

Summary:
This demonstrates how to query several independent IVF indexes with a trained index in common. This avoids to duplicate the coarse quantizer and metadata in memory.

On the Faiss side, it also implements a InvertedListIterator on top of the flat inverted lists, which can prove useful.

Reviewed By: junjieqi

Differential Revision: D56575887

fbshipit-source-id: cc3b26e952ee21f24b10169b5b614066600cf4b8

Summary:
`nullptr` is typesafe. `0` and `NULL` are not. In the future, only `nullptr` will be allowed.

This diff helps us embrace the future _now_ in service of enabling `-Wzero-as-null-pointer-constant`.

Reviewed By: palmje

Differential Revision: D56650318

fbshipit-source-id: 803ae62114c39143b65946f6f0387715eaf7f534

Summary:
This commit is the first in a series in an attempt to incrementally enable all jobs currenlty performed by CircleCI. It includes the main configuration files provided by GitHub team + 1 build.

Original PR: #3325

Reviewed By: junjieqi

Differential Revision: D56671582

fbshipit-source-id: c8a21cd69aabaf86134eb86753e90b1facf51bc3

Summary: GitHub checks

Reviewed By: junjieqi

Differential Revision: D56733297

fbshipit-source-id: fe5a2ca7c67f36a4fe986af78fb6dc8f4f843150

Summary:
Fixes #3379

Pull Request resolved: #3381

Reviewed By: junjieqi

Differential Revision: D56570120

Pulled By: kuarora

fbshipit-source-id: 758ea4ab866609d6dd5621e6e6ffda583ba52503

Summary: Migration to GitHub actions

Reviewed By: junjieqi

Differential Revision: D56745520

fbshipit-source-id: 5311a549842f19672ae574edaa8be3ea5a580dbc

Summary:
Pull Request resolved: #3405

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D56843276

fbshipit-source-id: 3d5c7ee9a36a783407dfdcc3574c377da5f9db78

Summary:
Pull Request resolved: #3406

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D56848895

fbshipit-source-id: 5a351534d9151369a9104314fee203657ac40043

Summary:
Pull Request resolved: #3407

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D56856565

fbshipit-source-id: d7400eb9cb7bd68e93a712af81c6cbb7e76e2400

… via GitHub Actions (#3409)

Summary:
Pull Request resolved: #3409

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D56917083

fbshipit-source-id: 93a2358ce5697b26aa40ced505f42c584fa8c46c

… availability (#3410)

Summary:
Pull Request resolved: #3410

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D56921925

fbshipit-source-id: 64e7a636b47d828110a6d413c8645e5343b578cb

Summary:
Pull Request resolved: #3411

Migration to GitHub

Reviewed By: kuarora

Differential Revision: D56923116

fbshipit-source-id: 1e2b493b0dd81ce850f2970e6d28c713f6a9927b

Summary:
Pull Request resolved: #3417

#3351

Reviewed By: junjieqi

Differential Revision: D57120422

fbshipit-source-id: e2e446642e7be8647f5115f90916fad242e31286

Summary:
Pull Request resolved: #3418

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D57133934

fbshipit-source-id: 255b7afbbb90cc966916cd900174833416b0bc51

Summary:
The code generated for function fvec_L2sqr generated by OpenXL do not perform as good as the codes generated by gcc on Power. The macros to enable imprecise floating point operation don’t cover Power with OpenXL. This patch adds the OpenXL compiler options for the PowerPC macros to achieve better performance.

Pull Request resolved: #3416

Reviewed By: asadoughi

Differential Revision: D57210015

Pulled By: mdouze

fbshipit-source-id: 6b838a2fa4d4996fe52c9f1105827004626fe720

…er libc (#3426)

Summary:
Pull Request resolved: #3426

GitHub Actions only supports Ubuntu 22 and newer and this change is necessary to enable CUDA builds to complete the migration.

Reviewed By: algoriddle

Differential Revision: D57261685

fbshipit-source-id: 34467f57426864ffa8b32f6018ccdc7bb4424b57

Summary:
Pull Request resolved: #3427

Migration to GitHub Actions

Reviewed By: algoriddle

Differential Revision: D57261696

fbshipit-source-id: d7b8c26259fd3de1cf59fc460f6af20185ceacfe

Summary:
Pull Request resolved: #3428

GitHub Actions currently does not support runners with AVX-512  but committed to add this support in early 2025. We will be running these on CircleCI until then. This placeholder build configuration will allow us to enable it with a 1-liner when the hosts are available.

Reviewed By: algoriddle

Differential Revision: D57261783

fbshipit-source-id: 1fb985a0c3dbb11851af63c95bde6494d25d0ac2

Summary:
This PR removes unneeded ARM NEON SIMD instructions for ScalarQuantizer.

The removed instructions are completely redundant, and I believe that it is a funky way of converting two `float32x4_t` variables (which hold 4 float values in a single SIMD register) into a single `float32x4x2_t` variable (two SIMD registers packed together).

Clang compiler is capable of eliminating these instructions. The only GCC that can eliminate these unneeded instructions is GCC 14, which was released very recently (Apr-May 2024).

mdouze

Pull Request resolved: #3430

Reviewed By: mlomeli1

Differential Revision: D57369849

Pulled By: mdouze

fbshipit-source-id: 09d7cf16e113df3eb9ddbfa54d074b58b452ba7f

Summary:
Pull Request resolved: #3442

fix install instruction for GPU + pytorch

Reviewed By: mlomeli1

Differential Revision: D57376959

fbshipit-source-id: 74caff960be7dbf8102e7593ce1485452a18de6e

Summary:
Pull Request resolved: #3432

Addresses the issue in #3173

for `IndexNNDescent`, I see that there is already interrupt implemented for it's [search](https://fburl.com/code/iwn3tqic) API, so I looked into it's `add` API.

For a given dataset nb = 10 mil, iter = 10, K =  32, d = 32 on a CPU only machine reveals that bulk of the cost comes from [nndescent](https://fburl.com/code/5rdb1p5o). For every iteration of `nndescent` takes around ~12 seconds, ~70-80% of the time is spent on `join` method (~10 seconds per iteration) and ~20-30% spent on `update` (~2 second per iteration). Adding the interrupt on the `join` should suffice on quickly terminating the program when users hit ctrl+C (happy to move the interrupt elsewhere if we think otherwise)

Reviewed By: junjieqi, mdouze

Differential Revision: D57300514

fbshipit-source-id: d343e0a292c35027ffdb8cbd0131e945b9881d63

Summary:
Pull Request resolved: #3439

LLVM-15 has a warning `-Wunused-but-set-variable` which we treat as an error because it's so often diagnostic of a code issue. Unused variables can compromise readability or, worse, performance.

This diff either (a) removes an unused variable and, possibly, it's associated code, or (b) qualifies the variable with `[[maybe_unused]]`, mostly in cases where the variable _is_ used, but, eg, in an `assert` statement that isn't present in production code.

 - If you approve of this diff, please use the "Accept & Ship" button :-)

Reviewed By: palmje, junjieqi

Differential Revision: D57344013

fbshipit-source-id: adf410139d2e6ca69a26ccdbff8511c9b7620489

…onda build recipes (#3440)

Summary:
Pull Request resolved: #3440

This change is required to unblock the migration to GitHub Actions. `cuda-toolkit` was only specified in the `libfaiss` package and it was not available in `faiss-gpu` or `faiss-gpu-raft`. This currently works on CircleCI because the runner image has CUDA toolkit of the needed version installed on the system and the build logic falls back to that but breaks on GitHub Actions because their runner images do not come with CUDA toolkit pre-installed.

Reviewed By: junjieqi

Differential Revision: D57371597

fbshipit-source-id: 8bededd53e2528f033fac797b296d74b47f9403e

Summary:
Pull Request resolved: #3443

Stabilize this file for clang-formatting versions 18.1.3 (VSCode) and 18.1.5 (our Github CI)

Reviewed By: junjieqi

Differential Revision: D57393650

fbshipit-source-id: 15170436bbd03194dbeaac1ef1130e20adc8c23e

)

Summary:
Pull Request resolved: #3441

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D57372738

fbshipit-source-id: 745b3c3f43c49045f8f5035e2af302ffa30d7755

Summary:
Pull Request resolved: #3414

#3355

A couple open questions:
- Given L2 was squared, I figured I would leave this one as squared as well?
- Also, wasn't sure if we wanted to return nan when present == 0 or -1?

Reviewed By: mdouze

Differential Revision: D57017608

fbshipit-source-id: ba14458b92c8b055f3bf2a871565175935c8333a

Summary:
Pull Request resolved: #3451

This is a first step to clean up the faiss codebase following T187322081

Reviewed By: junjieqi

Differential Revision: D57448335

fbshipit-source-id: c9760d01479d3352b786bbcf2015251e7a7168d6

Summary:
Pull Request resolved: #3452

Delete all remaining print within the Tests to improve the readability and effectiveness of the codebase.

Reviewed By: junjieqi

Differential Revision: D57466393

fbshipit-source-id: 6ebd66ae2e769894d810d4ba7a5f69fc865b797d

Summary:
Pull Request resolved: #3454

Removing commented out lines and adding proper descriptions and comments where appropriate.

Reviewed By: junjieqi

Differential Revision: D57501602

fbshipit-source-id: 0202ff73b7a83158808affba9b98b96dff569457

Summary:
Pull Request resolved: #3455

Code quality control by reducing the number of prints

Reviewed By: junjieqi

Differential Revision: D57502194

fbshipit-source-id: a6cd65ed4cc49590ce73d2978d41b640b5259c17

…3459)

Summary:
Pull Request resolved: #3459

When building with CUDA 11.4.4, CMake does not properly include files under Conda environment. This workaround flattens the include sub-directories in to the include root. It will unblock us for now while we are looking for a fix through CMakeLists files or figure out if it's a CMake bug and it gets fixed.

Reviewed By: junjieqi

Differential Revision: D57545169

fbshipit-source-id: 9cbdd0866e00e899cc889930a59448da55d873c2

…#3460)

Summary:
Pull Request resolved: #3460

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D57545637

fbshipit-source-id: 8ee970e5642ae9354455d60d84019d4217884d3a

Summary:
Pull Request resolved: #3461

Migration to GitHub Actions

Reviewed By: junjieqi

Differential Revision: D57545814

fbshipit-source-id: a3d08f8cf516ce26f8f16892eceef9e36bfe9f05

Summary:
Pull Request resolved: #3463

Satisfying linter warnings from a previous change

Reviewed By: junjieqi

Differential Revision: D57581364

fbshipit-source-id: 9e9b7f963a27d2da54d0e85390cce2f9f773c502

Summary:
Pull Request resolved: #3464

Migration to GitHub

Reviewed By: algoriddle

Differential Revision: D57593494

fbshipit-source-id: 726159b553d5544efcdfa064f5a82fd51ed793e9

Summary:
Pull Request resolved: #3466

Flattening Conda include directories breaks CUDA 11.4.4 build on Ubuntu 20 / v5 kernel. This change updates the logic to only flatten includes on Ubuntu 22 / v6 kernel (aka as running on GitHub Actions runners).

Reviewed By: algoriddle

Differential Revision: D57602154

fbshipit-source-id: 00c14ca7c64644b8b86483ac6b4d40c6d8f12372

Summary:
Pull Request resolved: #3467

1. Cron is scheduled to execute at 1:10am UTC. This is per GitHub's recommendation to avoid hotspots. The docs mention that when GH backend gets overloaded, they can drop scheduled jobs which we want to avoid so we scheduled off hour, off midnight.
2. The plan is to let these nightlies run once and, if successful, then disable them in GitHub UI to perform validation. Also disable if things break and need to be fixed of course.

Reviewed By: algoriddle

Differential Revision: D57602833

fbshipit-source-id: 4f4d9abbaa5ed3d1edb024ea4dd3f87aa78dd9b5

Summary:
Pull Request resolved: #3465

This commit include python version of tutorial for FastScan. It includes all the parameters enabled within PQFastScan.

Reviewed By: junjieqi

Differential Revision: D57594044

fbshipit-source-id: cb12679b6fc241a654b9545c5bc7bd0517aa1813

Summary:
'I' was printed twice and 'D' (distance vector) was not printed. Fixed.

Pull Request resolved: #3433

Reviewed By: fxdawnn

Differential Revision: D57451544

Pulled By: junjieqi

fbshipit-source-id: fc17b3b467f8b2c4ad7d80b44866456d9146e530

Summary:
Pull Request resolved: #3468

This commit includes the tutorial for PQFastScan in the cpp environment.

Reviewed By: junjieqi

Differential Revision: D57631441

fbshipit-source-id: f5e17eee2a584ebfc9ff63868d741d0da6b3b413

Summary: Pull Request resolved: #3469

Reviewed By: junjieqi

Differential Revision: D57650807

fbshipit-source-id: 5e642a8140455e4a3f1f21afe2f06771462e61f4

Summary:
Pull Request resolved: #3470

Hour and minute values are swapped, the goal is to run is at 1:10am UTC.

Reviewed By: algoriddle

Differential Revision: D57654059

fbshipit-source-id: 23bcb42e5c95f731cd4713ad4691d0f475ed8ad2

Summary:
Pull Request resolved: #3473

Previous diff (D57602154) fixed the CircleCI version and the PR build version of GHA but not the nightly one.

Reviewed By: junjieqi

Differential Revision: D57680576

fbshipit-source-id: 39f49c20df824c915f536b1ed3ffc35db2907988

Summary:
Pull Request resolved: #3474

This commit focus on the cpp version of PQfastscan tutorial with index refinement by defining the k factor.

Reviewed By: junjieqi

Differential Revision: D57680905

fbshipit-source-id: 980c2990172f24ec9a4f870685e706195883408f

Summary:
mdouze Please let me know if any additional unit tests are needed

Pull Request resolved: #3444

Reviewed By: algoriddle

Differential Revision: D57665641

Pulled By: mdouze

fbshipit-source-id: 9bec91306a1c31ea4f1f1d726c9d60ac6415fdfc

Summary:
Pull Request resolved: #3475

Migration to GitHub

Reviewed By: algoriddle

Differential Revision: D57707064

fbshipit-source-id: 17f0a97028007f3664faa5b6b2c269f50bcdf39e

Summary:
Pull Request resolved: #3476

The long tail will still be the RAFT build but we can save on cost if the build has errors by incurring ~10m penalty added by waiting for the basic cmake build to complete. Both GPU and RAFT builds will start together so this will take less time overall to complete.

Reviewed By: algoriddle

Differential Revision: D57707298

fbshipit-source-id: 3589842e9bda9ebca9b25e089e6177fe96b6a0f5

Summary:
Pull Request resolved: #3477

AVX-512 must remain on CircleCI until GitHub provides runners with AVX-512 support (ETA: Q1 2025).

Reviewed By: algoriddle

Differential Revision: D57707621

fbshipit-source-id: e8a0885f8363cf8f20854cccca3ec0adc946362b

….h (#3479)

Summary:
Pull Request resolved: #3479

`-Wextra-semi` or `-Wextra-semi-stmt`

If the code compiles, this is safe to land.

Reviewed By: palmje

Differential Revision: D57632759

fbshipit-source-id: 48bc23e87b3f518182085124c4c8e68ddbb3ca8f

Summary: Pull Request resolved: #3450

Reviewed By: mdouze

Differential Revision: D57708412

Pulled By: junjieqi

fbshipit-source-id: 9540b7e60d8b2b39e0ca92423d2a305fab2a17e6

Summary:
Small Raft related modification to StandardGpuResources:
if the stream for a particular device is modified by a user, delete the Raft handle for that device. On any subsequent call to `getRaftHandle(device)`, a new raft handle with the updated stream will be created.
Closes #3424

Pull Request resolved: #3435

Reviewed By: ramilbakhshyiev

Differential Revision: D57640976

Pulled By: junjieqi

fbshipit-source-id: 41e2898a39250b7e52e920b71e819fc21ca9fc85

…cy measurement (#3480)

Summary:
Pull Request resolved: #3480

This tutorial summarize the methods to construct different indexs for PQFastScan refinement. It shows how the choice can impact on accuracy.

Reviewed By: junjieqi

Differential Revision: D57799598

fbshipit-source-id: a75c52c60a5217366f3361676da8f03f0c4a9feb

Summary:
Rebased branch for #3420

Pull Request resolved: #3481

Reviewed By: mdouze

Differential Revision: D57830230

Pulled By: junjieqi

fbshipit-source-id: a93fb3cc53f11245faec891a9590b5e849dbf3b9

Summary:
simple math:
| **input n** | **input nshards** |  shard_size | idx | i0 | ni |
| -- |-- |-- |-- |-- |-- |
| 19 | 6 | 4 | 5 | 20 | **-1** |
| 1000 | 37 | 28 | 36 | 1008 | -8 |
| 1000 | 64 | 16 | 63 | 1008 | -8 |

root cause:
integer cause precision loss, `idx * shard_size` overflows, because `(n + nshards - 1) / nshards` is roundup

my solution:
each shard takes at least  `base_shard_size = n / nshards`, then `remain = n % nshards`, we know `0 <= remain < nshards`, next, assign those remain vectors to first `remain` shards, i.e. first `remain` shards take one more vector each.
```c++
auto i0 = idx * base_shard_size;
if (i0 < remain) {
  // if current idx is one of the first `remain` shards
  i0 += idx;
} else {
  i0 += remain;
}
```
simplify above code: `i0 = idx * base_shard_size + std::min(size_t(idx), n % nshards);`

Pull Request resolved: #3374

Reviewed By: fxdawnn

Differential Revision: D57867910

Pulled By: junjieqi

fbshipit-source-id: 7e72ea5cd197af4f3446fb7a3fd34ad08901dbb2

Summary:
Pull Request resolved: #3422

Found vec_io failing when running some benchmarking.
There is no such field named big_endian in sys. So, reverting it to original field byteorder

Reviewed By: algoriddle

Differential Revision: D56718607

fbshipit-source-id: 553f1d2d6bc967581142a92282e534f3f164e8f9

Summary:
Pull Request resolved: #3423

Adding small fixes to run experiments from fbcode.
1. Added buck target
2. Full import path of faiss bench_fw modules
3. new dataset path to run tests locally as we can't use  an existing directory ./data in fbcode.

Reviewed By: algoriddle, junjieqi

Differential Revision: D57235092

fbshipit-source-id: f78a23199e619b640a19ca37f8b52ff0abdd8298

Summary:
Pull Request resolved: #3487

Sometimes it is not useful to serialize the storage index along with a HNSW index. This diff adds a flag that supports skipping the storage of the index.

Searchign and adding to the index is not possible until a storage index is added back in.

Reviewed By: junjieqi

Differential Revision: D57911060

fbshipit-source-id: 5a4ceee4a8f53f6f746df59af3942b813a99c14f

Summary:
Pull Request resolved: #3494

This tasks focus on the refine index construction tutorial with different index refinement on fp16/sq8 quantization. The python version was added a while ago.

Reviewed By: junjieqi

Differential Revision: D58161983

fbshipit-source-id: 1c598fe612b5dee3952c5f7398e6802e117f141d

Summary:
Adding build folder to gitignore, so that they don't show up in the commit tree while building from source

Pull Request resolved: #3492

Reviewed By: junjieqi

Differential Revision: D58171359

Pulled By: asadoughi

fbshipit-source-id: b0efed348769328a3bdbcc13098dcb84cadb6c4f

Summary:
Depends on #3084

Pull Request resolved: #3252

Reviewed By: junjieqi

Differential Revision: D57971948

Pulled By: mdouze

fbshipit-source-id: 4371f4d136eeceb59568593f98a6ae9163a768ba

Summary:
Pull Request resolved: #3502

include probably added by vscode

Reviewed By: mengdilin

Differential Revision: D58411537

fbshipit-source-id: 3035f690d26decc937fb492c54ffa2f974ee2db8

Summary:
Add support for `use_raft` in the torch version of `knn_gpu`. The numpy version already has this support, see https://github.com/facebookresearch/faiss/blob/main/faiss/python/gpu_wrappers.py#L59

Pull Request resolved: #3509

Reviewed By: mlomeli1, junjieqi

Differential Revision: D58489851

Pulled By: algoriddle

fbshipit-source-id: cfad722fefd4809b135b765d0d43587cfd782d0e

Summary:
Pull Request resolved: #3512

Pull Request resolved: #3510

GitHub hosted runners some with the build-essentials package pre-installed, self-hosted runners on AWS do not have this package. This made it all steps other than the `all targets` one fall back to the system executables which unintentially worked on GitHub hosted runners but not on the self-hosted ones. This diff fixes it by pulling the line that adds conda bin to path early in the cmake build action.

Reviewed By: asadoughi

Differential Revision: D58513853

fbshipit-source-id: 23e95459e0031c96bd142515db07d1b700d713cf