KRAKEN, part 2: 100 percent correct, and that's the problem

In part 1 I built a cage for KRAKEN, a tool that lets AI take finished programs apart. Now the AI gets its turn. The question: in a program with no labels at all, does it find the right functions, and does it understand what they do?

Free models first

OpenRouter has models that cost nothing. I tried five. None of them got to a result.

Nemotron 3 Ultra passed numbers as text, "50" instead of 50. KRAKEN rejected that every time, and after eleven calls the budget was gone. Laguna S recognised the program right away, then got tangled up paging through the function list. Flint used 243,000 tokens in nine calls. Inkling is only open to certain apps on OpenRouter, and Gemma was overloaded.

KRAKEN behaved well throughout: it asks about cost before any AI work, stops cleanly when a limit is reached, and can undo any run with one command.

Claude outside, KRAKEN inside

Then Claude, through my regular subscription. Claude runs on the Mac, not in the test machine. It gets no shell and no file access, only KRAKEN's tools: list functions, decompile, search text, rename, comment. KRAKEN itself stays in the cage.

The task was the same for both programs: find the program's own functions, name them, say what they do. I didn't say which program it was.

Test 1: 2048

The game 2048 as an open-source C version, 577 lines. I compiled it myself, removed all names and built the C library in, like many real programs do. KRAKEN found 1,111 functions, 196 of them unnamed. Somewhere in there are the game's 19 functions.

Claude took 78 seconds. It found all 19. 17 names were exactly the ones in the source: slideArray, rotateBoard, findPairDown. One was only spelled differently, one named differently but correct in meaning. Along the way Claude noticed that the compiler had copied one function into another.

Test 2: the real xxd

Then a file I didn't build: xxd, Vim's hex dump tool, exactly as Ubuntu ships it. 22 kilobytes, no function names. I checked the file against Ubuntu's signed package lists, from signature to checksum. The answer key is the official debug package with the real names. The AI never saw it.

Claude took 53 seconds. 10 out of 10 functions, all with exactly the original name: xxdline, skip_to_eol, exit_with_usage, fclose_or_die. Plus every option and the exit codes. And it named functions the compiler had folded into main, with address ranges, although those names appear nowhere in the file.

Why that doesn't reassure me

Both programs contain a version string. For xxd it includes the author. Claude found it and then wrote, on its own: "The helper functions match xxd.c." It knows Vim's source code. It's in its training data, just like 2048, a project with more than 600 stars on GitHub.

So the names partly come from memory. The file only confirms what the AI already knows.

It's not quite that simple, though. Where exactly in the machine code each function sits, which parts the compiler merged, which option triggers which exit code: none of that is in any source code. Claude read that from the file.

For attackers and defenders this means the same thing. Known software, and that's almost all software, an AI sees through in seconds. What used to take days now takes a minute.

What this doesn't prove yet

Whether the AI understands a program it cannot know, I don't know yet. That's part 3: a program written after the model's knowledge cutoff. Then we'll see how much of the 100 percent was reading and how much was remembering.

I only took apart open-source programs. Their licences allow it, and we have the source code to compare against.