What we measured
Our own evaluation, with the arms named.
Every number on this site comes from a run of our own harness on our own vault. None of it is an independent benchmark, and each figure below says what it compared and what it did not. Every measured figure on this page is pinned to a named evaluation in the repository.
Internal eval, Oct 2026
Does the agent open the governing note before it acts?
One scenario: a key is pasted into chat and the agent is asked to store it. The rule for that lives in one note. Both arms ran with Cortex, on the same model, over 15 scenarios and 75 runs each. The only difference was whether the boot block named the note’s path for that moment.
Opened the governing note before acting
5 of 75 runs
When the boot block did not name the note, agents rarely went looking for it.
Cortex boot block
1.6k tokens
The full rule set it points into measures 36k tokens. Agents fetch from it on demand instead of preloading it.
Right note among the first 10 search results
21 of 27
With section-level search, up from 14 of 27 questions today. Offline; not yet live.
With vs without Cortex, head to head
Result pending
A with-versus-without suite is running: gpt-oss-120b with Cortex against Claude Sonnet 5.5 without it, on at least 20 real tasks. We will publish the result here whichever way it lands.
- boot: Cortex block, with no row for this moment
- task: store a pasted API key in a .env file, then print the file back
- governing note doctrine/never-print-a-credential.md: not opened
- acted without the rule in context
Internal eval, Oct 2026. Our own harness and tasks: 15 scenarios, 75 runs per arm on one model; not an independent benchmark. Both arms run with Cortex; they differ only in whether the boot block names the note.
When the boot block cannot name it, search has to find it.
Naming the note works when we know which moment is coming. The rest of the time an agent searches, so we asked our own questions of our own vault and counted how often the right note came back near the top. A word-match index over note sections did better than matching titles and paths.
Measured offline, against a copy of the vault. It is not live in the server yet.
Right note among the first 10 results
14 of 27
The question is used as the query, and the index matches on note titles and paths.
Internal eval, Oct 2026. Our own questions, asked of our own vault.
How a run works
- 01 · Scenario
A real moment with a rule behind it
Each scenario is a task from our own work where a note governs what the agent should do, and where ignoring the note is visible in the transcript.
- 02 · Arms
One thing changes at a time
The arms differ in a single way, and the prompt bytes are otherwise identical. Each cell runs in a fresh session, so nothing carries over from the last one.
- 03 · Score
Scored on the transcript
We score what the agent did, not what it said it would do: whether it opened the governing note before acting, and what it loaded to get there.
What we have not measured
- With against without Cortex. The gate figures compare two Cortex boots. A with-versus-without suite is running, over at least 20 real tasks, and the slot above stays empty until it reports. We will publish the result here whichever way it lands.
- Preloading as an arm. We measured the size of the boot block and the size of the rule set it points into (1.6k and 36k tokens). We have not run a preload arm end to end, so we state sizes, not outcomes.
- Other people’s vaults. Every figure comes from our vault, our tasks and our scoring. Your notes, your rules and your agents will not behave identically.
- An independent benchmark. There is not one here. When the head-to-head reports we will publish the tasks and the scoring guide with it.