Project 10Read · experimentPython · Docker · rules fixed in advance

Can fly-brain ideas make search and memory faster?

The fly's sense of smell inspired two well-known computing tricks. FlyHash turns each item into a short tag of active cells, so that similar items get similar tags and can be found quickly. The fly Bloom filter is a memory-saving way to check whether something has been seen before. This experiment tested them where they had never been tested: against the methods real systems use, against the fly's actual wiring, and on the standard speed-against-accuracy benchmark for search. Mostly, they lost.

Memory filter

Wins only at budgets where no method works

Real wiring

No better than scrambled wiring, unless missing data are filled in differently

Skewed questions

The prediction was contradicted

Search speed

About 1 question for every 200 the best index answers

Background

In the fly, a few dozen odour channels feed a much larger layer of cells, and each of those cells listens to only a handful of channels. Only the most strongly driven cells stay active, so each smell becomes a small set of active cells. In 2017 that design was turned into FlyHash, a method for finding similar items; in 2018 it was extended into DenseFly and into the fly Bloom filter. This experiment had four parts, each with its outcome stated in advance: a search benchmark, a memory-filter contest, a test of the fly's real wiring, and a test of one idea that could have rescued it.

The benchmark asks each method to find, for 10,000 questions, the 10 most similar items in a large collection, and it measures two things: how many of the 10 true answers come back (accuracy) and how many questions are answered per second (speed). It was run on two standard collections: SIFT, a set of one million image descriptors, and GloVe, a set of 1.2 million word vectors. The fly methods were up against four conventional ones: HNSW (hierarchical navigable small world), a graph-based index; ScaNN, Google's search library; and two versions of IVF-PQ (inverted file with product quantisation) from FAISS, Meta's similarity-search library. Every method was given its full published or registered range of settings.

Speed against accuracy for six methods on SIFT and GloVe. The four conventional methods run at thousands to tens of thousands of questions per second; FlyHash at about 100; DenseFly's line ends at accuracy 0.525 and 0.420.Speed against accuracy for six methods on SIFT and GloVe. The four conventional methods run at thousands to tens of thousands of questions per second; FlyHash at about 100; DenseFly's line ends at accuracy 0.525 and 0.420.
Higher is faster and further right is more accurate. At the same accuracy, the fly-inspired search methods answer about one question for every two hundred the best conventional method answers. One of them, DenseFly, never reaches the 90% accuracy level at all: its line stops at 52.5% on SIFT and 42.0% on GloVe.

Data: export.csv (every exported setting; each line joins only that method's best points) · trackB2_summary.json · report, part B2

The precise numbers at 90% accuracy
CollectionFlyHash, per secondBest conventional, per secondFlyHash's share
SIFT9221,131 (ScaNN)0.43%
GloVe5410,556 (ScaNN)0.52%

That is one question for every 231 on SIFT and every 194 on GloVe. FlyHash never reaches 95% on GloVe. Each method ran on one pinned processor core in the same container setup. The fly methods had at most as many settings as the most-tuned conventional method: 54 for FlyHash, 36 for DenseFly, 81 for HNSW.

Why FlyHash is slow

The fly methods were written in Python with the numpy library, and the conventional ones in C++, so part of the speed gap is the programming language. To separate the method from the language, the fly code counted how much work each question takes, which does not depend on the language.

Bars: list entries FlyHash examines per question for each of its 54 settings on each collection, against the database size. Every setting that reaches 90% accuracy is above the database line.Bars: list entries FlyHash examines per question for each of its 54 settings on each collection, against the database size. Every setting that reaches 90% accuracy is above the database line.
Each bar is one FlyHash setting, and the dashed line is the number of items in the database. Every setting that reaches 90% accuracy (red) examines more candidate entries per question than the database contains: 1,278,276 for the one-million-item SIFT and 4,884,445 for the 1.2-million-item GloVe. That is a property of the method, not of the programming language it was written in, so a faster implementation would make each step quicker but would not reduce the number of steps.

Data: flyhash_candidates_touched.csv, copied from the 108 FlyHash result files · export.csv · collection sizes

An entry is one appearance of an item in one of FlyHash's lookup lists, and an item appears in several lists, which is how a question can touch more entries than there are items. A version written in C++ would have to be about 230 times faster on SIFT, and 195 times on GloVe, just to match the best conventional method. Whether it would still lose was not tested, but it would still touch more entries per question than the database holds items.

The memory filter

A familiarity filter answers “have I seen this, or something very like it, before?” using a fixed amount of memory per stored item. The fly Bloom filter was compared with the methods production systems use: a store of short SimHash fingerprints (compact codes that similar items share), MinHash (a sketch for estimating overlap between items), and a count-min sketch, a compact table of counters. The datasets were fly odour responses, SIFT image descriptors, MNIST handwritten digits, random data, and news articles from the New York Times. The score measures how well each method's rating ranks items by how new they really are.

Top: the fly filter's score minus the best conventional method's at 3, 10, 30 and 100 bits per item, for five datasets. Bottom: share of familiar items each method misses at a 1% false-alarm rate; at 10 bits and below no method misses fewer than 83.9%.Top: the fly filter's score minus the best conventional method's at 3, 10, 30 and 100 bits per item, for five datasets. Bottom: share of familiar items each method misses at a 1% false-alarm rate; at 10 bits and below no method misses fewer than 83.9%.
In the top panel, points above zero mean the fly filter was better: it leads at 3 and 10 bits of memory per item and mostly trails at 30 and 100. The bottom panel shows why the lead does not help: at 10 bits and below, every method misses more than four out of five familiar items once false alarms are held to 1%. The fly method is better at the smallest memory budgets, and at those budgets no method is usable for the job. Where the task is actually solvable, a simple conventional fingerprint wins.

Data: trackA_summary.json · report, part A

Two exceptions are visible in the top panel. On random data the fly filter still leads slightly at 30 bits. The odour dataset has only 110 items, too few to tell either way at any budget. The fly filter's wins also cost at least 8 times more arithmetic per item than the method it beat, and its cost per item grows as more items are stored.

Does the fly's real wiring help?

FlyHash usually connects its input channels to cells at random. The fly's own connections are not random, so a natural question is whether using the real wiring, taken here from MaleCNS v1.0, the map of the male fruit fly's central nervous system, makes the tags better. A 2026 preprint, a paper not yet peer reviewed, by a single author, found it made them slightly worse: −2.06% against a shuffled version. This part repeated that test independently, on odour responses from DoOR, the Database of Odorant Responses, and compared the real wiring against four kinds of scrambling, each keeping more of the real structure.

Dot and range chart, left and right hemisphere: the real wiring's score against four scrambled controls, with missing odour data set to zero and filled with averages. With averages, the left hemisphere beats the standard shuffle by 3.38%.Dot and range chart, left and right hemisphere: the real wiring's score against four scrambled controls, with missing odour data set to zero and filled with averages. With averages, the left hemisphere beats the standard shuffle by 3.38%.
Look first at the circles, where missing odour measurements are set to zero: none reaches the shaded zone where the real wiring would be at least 2.5% better than a scrambled version. Now the squares, where the gaps are filled with averages: the left side of the brain jumps to 3.38% better than the standard shuffle. Whether the fly's real wiring beats a scrambled version depends on how the missing 28.7% of the odour measurements is filled in. That dependence is itself the finding.

Data: report, part B1, table “Odours: BIO against each null”, rows w5_L and w5_R

On the same data the preprint used, this experiment measured −2.09%, which reproduces its result. The main analysis, decided in advance, used zero-filling, so the official verdict is that the preprint was replicated: the real wiring gives no benefit as large as 2.5%. The strictest scrambling, which also keeps how strongly inputs converge on each channel, comes closest to the real wiring, which suggests that convergence pattern is what pulls the real wiring's score down; that reading was not part of the plan.

One idea that could have rescued it

The real wiring over-connects some odour channels. The experiment's author proposed that this might pay off when questions are concentrated on exactly those channels, the way natural smells might be. The idea and its definition were written down and fingerprinted before any number existed, and then tested.

Line chart, left and right hemisphere: real wiring's advantage over its shuffle as questions concentrate. Left, matched: falls to −7.17%. Right, matched: rises to +1.62%, below the mismatched line and the +2.5% band.Line chart, left and right hemisphere: real wiring's advantage over its shuffle as questions concentrate. Left, matched: falls to −7.17%. Right, matched: rises to +1.62%, below the mismatched line and the +2.5% band.
The idea was that the fly's unusual wiring might pay off when questions are concentrated on the channels it over-connects, so the solid line should climb into the shaded band. The opposite happened. On the left side of the brain it fell to 7.17% worse; on the right it rose only to 1.62% better, below the band and below the dashed line for the opposite channels.

Data: trackC_summary.json · report, part C

The precise numbers: real wiring against the standard shuffle
ComparisonLeft hemisphereRight hemisphere
Missing data set to zero−0.55% (−1.30% to +0.20%)−2.38% (−3.16% to −1.60%)
Missing data filled with averages+3.38% (+2.61% to +4.15%)−0.63% (−1.43% to +0.19%)
Strongest concentration, matched channels−5.54% (−5.84% to −5.23%)+1.62% (+1.35% to +1.88%)
Strongest concentration, opposite channels+0.02% (−0.32% to +0.38%)+2.34% (+1.99% to +2.68%)

Ranges are 95% ranges. Connections count only with at least 5 synapses. The lowest point on the left, −7.17%, is at moderate concentration.

What this does not show

How it was run

The plan, each part's possible verdicts and the analysis code were fixed before the first measurement and signed off; every later change is one of 16 logged amendments. The search benchmark ran in the standard benchmark harness, in Docker containers with one processor core each.

Sources: synthesis, part A, parts B1 and B2, part C, preregistration.

← 09 Forward-only trainingAll results →