Project 10Read · experimentPython · Docker · rules fixed in advance
The fly's sense of smell inspired two well-known computing tricks. FlyHash turns each item into a short tag of active cells, so that similar items get similar tags and can be found quickly. The fly Bloom filter is a memory-saving way to check whether something has been seen before. This experiment tested them where they had never been tested: against the methods real systems use, against the fly's actual wiring, and on the standard speed-against-accuracy benchmark for search. Mostly, they lost.
In the fly, a few dozen odour channels feed a much larger layer of cells, and each of those cells listens to only a handful of channels. Only the most strongly driven cells stay active, so each smell becomes a small set of active cells. In 2017 that design was turned into FlyHash, a method for finding similar items; in 2018 it was extended into DenseFly and into the fly Bloom filter. This experiment had four parts, each with its outcome stated in advance: a search benchmark, a memory-filter contest, a test of the fly's real wiring, and a test of one idea that could have rescued it.
The benchmark asks each method to find, for 10,000 questions, the 10 most similar items in a large collection, and it measures two things: how many of the 10 true answers come back (accuracy) and how many questions are answered per second (speed). It was run on two standard collections: SIFT, a set of one million image descriptors, and GloVe, a set of 1.2 million word vectors. The fly methods were up against four conventional ones: HNSW (hierarchical navigable small world), a graph-based index; ScaNN, Google's search library; and two versions of IVF-PQ (inverted file with product quantisation) from FAISS, Meta's similarity-search library. Every method was given its full published or registered range of settings.
Data: export.csv (every exported setting; each line joins only that method's best points) · trackB2_summary.json · report, part B2
| Collection | FlyHash, per second | Best conventional, per second | FlyHash's share |
|---|---|---|---|
| SIFT | 92 | 21,131 (ScaNN) | 0.43% |
| GloVe | 54 | 10,556 (ScaNN) | 0.52% |
That is one question for every 231 on SIFT and every 194 on GloVe. FlyHash never reaches 95% on GloVe. Each method ran on one pinned processor core in the same container setup. The fly methods had at most as many settings as the most-tuned conventional method: 54 for FlyHash, 36 for DenseFly, 81 for HNSW.
The fly methods were written in Python with the numpy library, and the conventional ones in C++, so part of the speed gap is the programming language. To separate the method from the language, the fly code counted how much work each question takes, which does not depend on the language.
Data: flyhash_candidates_touched.csv, copied from the 108 FlyHash result files · export.csv · collection sizes
An entry is one appearance of an item in one of FlyHash's lookup lists, and an item appears in several lists, which is how a question can touch more entries than there are items. A version written in C++ would have to be about 230 times faster on SIFT, and 195 times on GloVe, just to match the best conventional method. Whether it would still lose was not tested, but it would still touch more entries per question than the database holds items.
A familiarity filter answers “have I seen this, or something very like it, before?” using a fixed amount of memory per stored item. The fly Bloom filter was compared with the methods production systems use: a store of short SimHash fingerprints (compact codes that similar items share), MinHash (a sketch for estimating overlap between items), and a count-min sketch, a compact table of counters. The datasets were fly odour responses, SIFT image descriptors, MNIST handwritten digits, random data, and news articles from the New York Times. The score measures how well each method's rating ranks items by how new they really are.
Data: trackA_summary.json · report, part A
Two exceptions are visible in the top panel. On random data the fly filter still leads slightly at 30 bits. The odour dataset has only 110 items, too few to tell either way at any budget. The fly filter's wins also cost at least 8 times more arithmetic per item than the method it beat, and its cost per item grows as more items are stored.
FlyHash usually connects its input channels to cells at random. The fly's own connections are not random, so a natural question is whether using the real wiring, taken here from MaleCNS v1.0, the map of the male fruit fly's central nervous system, makes the tags better. A 2026 preprint, a paper not yet peer reviewed, by a single author, found it made them slightly worse: −2.06% against a shuffled version. This part repeated that test independently, on odour responses from DoOR, the Database of Odorant Responses, and compared the real wiring against four kinds of scrambling, each keeping more of the real structure.
Data: report, part B1, table “Odours: BIO against each null”, rows w5_L and w5_R
On the same data the preprint used, this experiment measured −2.09%, which reproduces its result. The main analysis, decided in advance, used zero-filling, so the official verdict is that the preprint was replicated: the real wiring gives no benefit as large as 2.5%. The strictest scrambling, which also keeps how strongly inputs converge on each channel, comes closest to the real wiring, which suggests that convergence pattern is what pulls the real wiring's score down; that reading was not part of the plan.
The real wiring over-connects some odour channels. The experiment's author proposed that this might pay off when questions are concentrated on exactly those channels, the way natural smells might be. The idea and its definition were written down and fingerprinted before any number existed, and then tested.
Data: trackC_summary.json · report, part C
| Comparison | Left hemisphere | Right hemisphere |
|---|---|---|
| Missing data set to zero | −0.55% (−1.30% to +0.20%) | −2.38% (−3.16% to −1.60%) |
| Missing data filled with averages | +3.38% (+2.61% to +4.15%) | −0.63% (−1.43% to +0.19%) |
| Strongest concentration, matched channels | −5.54% (−5.84% to −5.23%) | +1.62% (+1.35% to +1.88%) |
| Strongest concentration, opposite channels | +0.02% (−0.32% to +0.38%) | +2.34% (+1.99% to +2.68%) |
Ranges are 95% ranges. Connections count only with at least 5 synapses. The lowest point on the left, −7.17%, is at moderate concentration.
The plan, each part's possible verdicts and the analysis code were fixed before the first measurement and signed off; every later change is one of 16 logged amendments. The search benchmark ran in the standard benchmark harness, in Docker containers with one processor core each.
Sources: synthesis, part A, parts B1 and B2, part C, preregistration.