Project 08Read · experimentPyTorch · rules fixed in advance
A simulated robot has to keep a moving target in view while it steers around posts and walls. Its controller was wired with a 443-neuron circuit taken from the fly's connectome, the full map of the fly's nerve cells and their connections, and trained until it stopped improving. It was then compared with the same controller with the circuit taken out. The circuit did not clearly help. What it did do is more interesting: its loops act as a short memory, and that memory both helps and hurts.
The robot sees the world through a one-row strip of light sensors, like a single line of pixels. Two versions of its controller are compared here. One routes the signal through the fly circuit, which has loops: neurons feed back into each other, so the circuit's state carries a trace of what it saw a moment ago. The other, the direct connection, maps what the eye sees straight to the steering, with no loops and so no memory. Each version was trained 30 or 40 times from different random starting points and scored on fresh scenes it had never seen.
Data: stage1_headroom.md · stage1b_occlusion.md · g1.json · g1b.json
Share of evaluation episodes with at least one collision. Target visible: 1,000 fresh scenes, 30 training runs per version. Target hidden: 2,000 fresh scenes, 40 runs per version. A difference is called reliable when we can be 95% confident it is not zero.
| Collision, condition | Circuit | Direct | Circuit minus direct (95% range) |
|---|---|---|---|
| Post, visible | 13.2% | 13.3% | −0.1 points (−1.3 to +1.1) |
| Wall, visible | 7.4% | 4.5% | +2.8 points (+1.8 to +4.0) |
| Post, hidden | 18.2% | 21.4% | −3.2 points (−4.4 to −2.1) |
| Wall, hidden | 25.5% | 22.8% | +2.8 points (+1.4 to +4.1) |
In an earlier experiment, a controller built on this circuit did no better than the same controller with the circuit removed, and a hand-written classical tracker beat both: it succeeded in 92.5% of test scenes, against 70.9% for the fly circuit. One objection was that the circuit had been given a fixed, short training budget and might simply not have finished learning. This experiment trained every version until it stopped improving and asked again: does the circuit beat the direct connection?
Two tests were fixed in advance. The first used the normal task. The second hid the target for two spells of 1 to 2 seconds in every scene, the one situation where a memory should give the circuit a real reason to win. The circuit counted as clearly better only if it led by at least 3 points on the main score and we could be confident the lead was above zero. It counted as ruled out if we could be 95% confident its lead was below 3 points, on both the main score and the success rate. Only a clear win would have opened the next stage, which asks whether it is the fly's exact wiring that matters.
The main score is the share of time the robot kept the target in view at the right distance. A scene counts as a success when the robot kept it in view most of the time, never lost it for long, and hit nothing.
Target visible: ruled out. The circuit scored 88.9% against the direct connection's 91.0%, so it was 2.1 points behind. We can be 95% confident the true gap is between 0.7 and 3.5 points in the direct connection's favour. Its success rate was also lower.
Target hidden: unclear. The circuit scored 69.2% against 68.5%, 0.7 points ahead. We can be 95% confident the true difference lies between 0.5 points behind and 2.0 points ahead, so a lead could not be confirmed. Its success rate was 2.5 points higher, and that lead is reliably above zero, but its range reaches 4.1 points, above the 3-point bar, so a worthwhile benefit could not be ruled out either. After the first round came back unclear, the plan allowed one extension, to 40 runs per version and 2,000 test scenes. It stayed unclear, and that verdict is final.
The training runs varied more than planned for: up to about 1.86 times the planned spread, for the circuit's success rate. So this unclear result should not be read as weak evidence that there is no effect. It is a test that could not decide.
| Circuit minus… | Target visible | Target hidden |
|---|---|---|
| direct connection, main score | −2.1 points (−3.5 to −0.7) | +0.7 points (−0.5 to +2.0) |
| direct connection, success rate | −2.3 points (−4.0 to −0.5) | +2.5 points (+0.8 to +4.1) |
| two-signal summary, main score | +10.4 points (+7.0 to +14.3) | +10.0 points (+7.6 to +12.5) |
Ranges are 95% ranges from resampling training runs and scenes, 10,000 times. Main score with the target visible: circuit 88.9%, direct 91.0%. Hidden: 69.2% and 68.5%.
If the post advantage came from the fly's particular wiring, a scrambled copy of the circuit should lose it. Three rewired copies were trained alongside: the same neurons and the same number of connections, with the connections shuffled so that each neuron keeps its count of inputs and outputs.
Data: stage1b_occlusion.md, table “Every arm” · g1b.json
The rewired copies were trained 15 times each, against 40 for the real circuit and the direct connection, and they were compared only descriptively: no test was set in advance for them. On the main score none of them scored reliably worse than the real circuit, and one, rewired copy 3, did slightly better with the target visible: the real circuit was 1.8 points behind it, and we can be 95% confident that gap is above zero.
Before training, every connection's starting strength is set from its synapse count, the number of physical contacts between the two neurons. That turned out to matter on its own. Measured before any learning, with identical settings, the real circuit is quieter than every one of 64 scrambled versions of itself.
The earlier experiment's rewired copies, and the three in the figure above, started 108.8% to 186.0% more active than the real circuit. The planned next stage would have matched every scrambled circuit's starting activity to the real one before training, and also kept the unmatched versions, so the size of this effect would stay visible.
| Scrambling keeps… | Graphs | Extra output activity |
|---|---|---|
| each block's synapse strengths | 10 | +0.2% to +8.2% |
| cell-type and side blocks | 20 | +3.0% to +102.5% |
| cell-type blocks | 10 | +69.7% to +121.2% |
| each neuron's number of links | 10 | +20.2% to +108.1% |
| excitatory and inhibitory counts | 10 | +8.8% to +91.9% |
| earlier rewired copies | 3 | +108.8% to +186.0% |
| earlier random copy | 1 | +40.0% |
Mean output-neuron activity over 50 fixed scenes, averaged over 10 starting points, relative to the real circuit. The direct connection, which has no circuit to scramble, is not shown; it was 29.0% less active.
Data: stage1_headroom.md and stage1b_occlusion.md, tables “Convergence” and “Mean validation tracked fraction by step”
The next stage would have asked how much of the circuit's lead comes from the fly's exact wiring, by comparing it with 20 scrambled circuits, each trained 5 times. That only makes sense if there is a lead to explain.
Data: power.json, Stage 2 table · stage1b_occlusion.md, verdict table
“Reliably detect” means the planned test would catch a real difference of that size four times out of five. The plan did not say what to do after a final unclear result at the second test, so the next stage was not started.
The plan was written and fixed before any training run: the versions to compare, the number of runs, the fresh test scenes, the thresholds for a clear win and for ruling out, the single extension, and the analysis code, which was frozen and checked by its fingerprint before each analysis. Test scenes could only be scored through one guarded script. Every number on this page is rendered from the saved result files linked below; nothing is re-derived for the site.
Sources: first test report, second test report, preregistration, earlier experiment's report.