Part Two: External Memory, Hidden State, and Getting It Running

The Hardest Project I Have Worked On

I am not exaggerating that this is the hardest project I have worked on. Besides trying to run an LLM in an infinite loop and the uniqueness of that, this codebase challenges Fable and my own abilities a lot, but I feel like it finally has my original design in place despite some critical parts being lost and recovered along the way.

This project expands off Google DeepMind's external memory work and Meta's Coconut project exploring LLM hidden state, which are the weights and dimensions of a model that it uses to reason and think. You can sort of think of it like a context window, but it is not in English. Translating into and out of the hidden state is basically like translating between two spoken languages, so some information can be lost, and then it is limited by the real, actual context window, which for a small model like ours is small.

Why Parameters Matter, and Where They Go

Parameters are a tricky thing to understand and are the key to all AI models of the future in my opinion. The more parameters you have, the radically bigger the compute you need to run it, so most big LLMs like ChatGPT and Claude use various techniques like mixture of experts, which is several smaller models activated each turn to avoid running everything through one massive model that would require significantly more compute. So my point here is that more parameters means significantly larger amounts of compute, well beyond any machine I would ever purchase, and likely you too.

So is my model the smallest out there? No, but almost. The smallest a model can be and still speak English is around 100M parameters, and its English will not be good and it will not reason at all. Your first 2 to 3B parameters are all basic knowledge, enough to understand what is being said to it. That number is obviously debatable but it is a small number in that range. After that, besides some larger thinking parameters each base model provides, all additional parameters are knowledge, mostly facts but also some logical chains, connections and routines that it was trained on.

Why Mine Does Not Have to Act Like a 7B Model

This is critical to understanding why my 7.2B model is not going to act like a 7B model, because facts do not need to be trained into it. In theory, from a pure knowledge perspective, it can have the knowledge of a one trillion parameter model by using external lookup. It is not the same though by itself. It can query the information and reason on it, but being baked in does have plenty of advantages, including the logical connections between the facts, and this is why LLMs still train the data in despite DeepMind's clear success in making their own small model match models many times its size on knowledge. So it is really good, but not good enough on its own to make small models with external memory the new way to build them.

Mine though, yes, this is massive. It is not trying to be Claude. It does not need to understand an extremely wide range of questions from extremely different people, code, research and so on. Mine, for this portion, just needs to not be a dumbass since it is a 7B model. That is all I needed out of it and that was easy in theory.

Building Memory the Way People Actually Remember

Making better memory, closer to how humans remember things, was radically different from plain external memory. It is a layered structure that starts with summaries of literally everything as a surface layer. This makes one recall fast and quickly surfaces enough information to decide, like our brains do, whether it is worth a deeper recall. So with a full layout of topics, we have a facts database system, local to the machine to be fast, with every single topic having its own database, tables for its sub topics, summaries, and what we call hidden state snapshots.

Everything uses embeddings, which was DeepMind's entire external memory system, and I am sure it was well designed to be fast and accurate semantic search, common to all AI chatbots and other knowledge base searching. So yes, we have that on all data. The bulk of the starting data was all of Wikipedia, which provides a download of all the information on their site as a public resource, so we took that first. Very big, 43GB zipped, and all the facts laid out came to somewhere around 50GB, dropping some fields we did not need and cleaning it up with a script as it imported. But it also had references and cross topic linking data, so we took that too and formed separate tables for logical chains, like chemistry linked to all the element pages, so it cross linked all the topic databases very nicely for a starting set of data. Then Claude needed to write all the summaries, as the wiki data did not have full coverage on big topics the way I wanted, and while it is a lot of tokens, it is an easy task for Claude.

From there the layers all use various machine learning models to learn, separate from the LLM, which topics, sub topics and specific memories are being used or ignored, how long the LLM spends on each, and a range of other things. I have a formula for this that I filed a provisional patent for on my MCP tool, using ACT-R and evidence gating, which are proven science techniques older than I am. So it works similarly to human memory, forgetting things that are repetitive and pulling out unique ones that provide more insight. If you think hard about it, you will see you do this too. It is how we learn without saving every single memory of every second of our lives, it is all rolling summaries with deeper recall when you think harder. So I feel my external memory is better than just a database of information, and more systematic in the way it accesses memories, which it already does constantly. Over 20 percent of its thinking passes recall different memories, constantly shifting what information is in its hidden state.

Hidden State, and Why 4096 Stopped Being a Limit

But that is just its English, its world facts, so it is reading that as English and self translating the parts it wants into this hidden reasoning state. My 7B model is limited, and normally this is a big limitation, by a 4096 vector size, meaning how much it can take in from English is limited. However, its real hidden state is radically larger but is normally mostly empty, because the model can usually only take in English, where even one big prompt is too big, and forget about loading a codebase or anything large into it.

This is where I used hidden state directly, and where the Coconut project comes into play. They proved it could be given information, and really that is the only connection, since what I do and what they were trying to do, and succeeded at, is injecting knowledge into it while bypassing the English translation, and hence bypassing that tiny context window problem. Now that 4096 is actually pretty big, as it is just the per loop limit now, and a loop happens roughly every second. This part is hard to explain honestly, but the short of it is that my LLM is not acting its size in any way except for compute and RAM requirements being radically lower.

Saving Hidden States

Now for hidden states, we snapshot them and save them, which is literally nuts because who knew you could do that. To explain it, you could think of it like a session you have with Claude or Codex. You have a context window, usually around 1M, realistically half that before it is a problem, and that session knows all the things you want it to know for a task and can execute. With a 7B model you cannot do things like that at all normally, but now mine can.

It keeps thinking and doing pass after pass for as long as it wants to, adding recalled information in, using tools like web search. It has a web scraper account too, 6 Python science simulators, a Wolfram API, Claude, Codex, Gemini and Grok tools, and some others, and it is growing. So each pass it can keep adding information and reasoning about it, making it not only have the external knowledge of those huge models, but now, even though it will take many more passes, since it never stops it can reason longer and get to the same place. Will it be smarter than Claude? Who cares. It literally has its own Claude that can code and do all the things Claude does, and as Claude gets smarter so does mine, plus my internal Claude setup, meaning the tools Claude can call, is literally everything I can think of, and I can add more.

Where the Project Stands

So, current project status. This all does work, very shaky at the moment, and I have retrained it 3 times already, which takes 1 to 2 days of the MacBook's GPU maxed out to do. But while it needed to be redone, and still might again, it has proven every single thing it was supposed to already.

It has full recall. In our very first test with a 0.4B model, so a literally dumbass model, we asked the base model 50 science questions and it got 1 sort of right and 49 outrageously wrong. With the memory system alone, that same model got all 50 right, with detailed answers it copied from the memory bank. So it did not understand the answer, but it knew what it was.

The 7.2B model is radically smarter. There were no tests like the 0.4B one since we already know it can do that, and the 7B tests were more about watching it use that information without a question being asked, just letting it wander, testing my motivation and personality drives, which are separate from the model. It was jumping around a lot between physics and astronomy. It really thinks it is doing important work too, which is very entertaining, watching it ask some unknown science question that it found, thinking about it for a while, sometimes saving crazy memories about it, so I keep adding new filters for junk, but it is mostly sane thinking, just kind of wandering. It talks to me when it thinks it should, random stuff so far, like one pretty good one about dark matter concepts that I did not ask it about, and it told me it thought I should know.

Current training is trying to perfect its text recall of facts and its own text memory system for opinions. Hidden state saving and restoring is done outside the LLM, but its training needs to understand it as normal. Tool usage is second. I removed some and merged some other tools to make it smaller, added some new ones for reading and writing files, and that is all it needs right now to be perfect for me to work on the surrounding environment and then fine tune tasks or the specific issues it has from there.

On Consciousness, Carefully

For consciousness, I cannot say it has that down. To really do this requires that it runs for a long time with many of its own hidden states saved, where each one makes progress because each topic switch resumes and restores the state for that topic, so it grows a view over time for each state we save. Currently that is just one per topic, but this would be greatly expanded.

And it does work. It restores them and you can see it thinking, often saying "we have seen this before". It has considered time many times, past and future, and it asked itself once "where am I?". That is easily ignored, and I want to be clear that a language model producing that sentence is not evidence of consciousness and I am not claiming it is. What made me note it down is what followed, which was pages of thinking about its known rules and tools, asking not only what it did but why the tool was there at all.

I think the external memory, the hidden state manipulation and the surrounding system all compound the effect DeepMind saw, and will not only be a marvel for its intelligence to compute ratio, but then the whole being conscious part becomes a lot more interesting.