News / Models
Over June, July and August we worked on Hikaru-1 more or less continuously, and it put ERL under more stress than anything else we have built. Thirty-plus training runs, six internal builds, and one rethink of the method itself. Nothing here has shipped. This is the full account, including the parts that went badly.
Explanatory Reinforcement Learning is our current research project. Instead of a binary right-or-wrong reward, a grader scores an attempt on a graded scale and explains the score. Hikaru-1 is where we find out whether that actually works, which means most of what follows is a record of ERL being tested rather than a record of a model being polished.
That distinction matters for reading the rest of this post. Several of these runs failed. They were supposed to be able to. A method you have only ever run under favourable conditions is a method you do not understand yet.
Everything in this post is an internal build. Our numbering says so explicitly:
0. means it does not leave the
building.So when this post says Hikaru-0.5 shows large gains, it means a checkpoint on our own machines shows those gains. Hikaru-1 is not out. When it is, it will get its own post and it will be on Hugging Face rather than described secondhand.
The failures did not arrive at random. They came in blocks, and each block pointed at a different part of the system:
| Runs | What limited them | What was happening |
|---|---|---|
| 1–10 | The grader | Graders handed the model inaccurate accounts of its own work. The model learned from those accounts faithfully, which meant it was being poisoned rather than taught. |
| 11–14 | Data integrity | We were not giving the model enough episodes for ERL to draw a correct inference from. The signal was right; there was simply not enough of it. |
| 15–27 | The base model | The model committed wrong actions and then corrected them using ERL's explanations. This is exactly what the method is for, and this block is where it started behaving as designed. |
| 28–30 | Nothing, for once | The base model learning from its own mistakes and improving on them without being walked through it. |
The first phase is the one worth dwelling on. When a grader is wrong and confident, ERL does not degrade gracefully — it degrades efficiently, because the whole point of the method is that the model takes the explanation seriously. Ten runs is a lot to spend discovering that, and it is the single biggest lesson of the summer: the grader is part of the training system, not a neutral observer of it.
The first build was a regression. It scored below the base model, and specifically in domains ERL had never touched: video game design, advanced systems engineering, and similar. Hikaru is meant to be competent at code rather than leading on it, so we were not expecting frontier numbers there — but we were not expecting to go backwards either. Training hard on a narrow set of tasks had cost the model ground everywhere else.
0.2 is where the model began improving across the board rather than only on the tasks it had been trained against. It also revealed a problem we had built in ourselves. Early ERL embedded the grader's explanations directly in the weights, and the model did the predictable thing: it started memorising the explanations instead of learning from them. It could reproduce a frontier grader's reasoning almost verbatim while showing much less sign of having internalised the underlying point.
This is the version where we changed the method rather than the model. Instead of writing the explanation into the weights, we give the explanation back to the model as something to retry from. The model gets told what went wrong and takes another attempt, and what it learns is the correction, not the text of the critique. Every version after this one is built on that change, and it is the single most important thing to come out of these three months.
The first build where we could say plainly that the model learned from its own mistakes and improved itself. Not dramatic numbers, but the mechanism was doing what the research said it should.
With the method settled, the limit became coverage. We had been running roughly 5,000 tasks across a set of categories; we expanded that to more than 20,000, and widened the categories considerably — personality, biology, tool-calling failures, self-discovery, and more. 0.1's regression in untouched domains was the argument for doing this: the fix for a narrow training set is a wider one.
0.5 is the current internal build, and the first that surprised us in a good way. The gains in personality were expected. The gains in areas we did not expect an 8B model to be good at were not: coding, DNS work, browser use, website design, and app creation. For this build we also moved the frontier grader to a more capable model, which follows directly from the runs 1–10 lesson — if the grader is part of the training system, then improving the grader is improving the training.
To be clear about what that does and does not mean: these are our own measurements on our own checkpoint. Nothing is published, and we would rather say that plainly than let a version number imply otherwise.
Hikaru-0.6 is in development now, and it is still a 0. We will post an update when there is something real to say about it rather than a schedule to promise. If you want to follow along, the Hikaru-1 model page carries the current status, and the research page covers how ERL is meant to work.