News / Models
It hasn't been long since the last update on Deriva-1, and yet there is already a lot to talk about. Two small internal builds landed with little to show for them, the data mix changed in a serious way, and an early look at Deriva-0.6 suggests we are finally moving in the right direction again.
Since the last update we have shipped two small internal builds, Deriva-0.4 and Deriva-0.5, and neither moved the needle the way we had hoped. Both are real steps in the process; they are just not the kind of step that shows up on a leaderboard. What they did show us is exactly where the work still has to happen, which is the more useful finding, even if it is not the one we wanted.
With Deriva-0.4 the base model's issues shined through very
clearly. Overthinking is still the core problem in this
version, and it is a difficult one to handle, because how are
you supposed to judge what deserves more thinking and what
deserves less? Frontier models handle that sizing
automatically, and Qwen3.8-27B does not. It
will spend thousands of tokens on a one-line factual question
and a handful on something that genuinely needs depth, and
getting the model to consistently size its own thinking to
the task is harder than any of the other fixes we have made
so far.
0.4 also showed us how to improve. In previous versions we
had been using synthetic data from a single model, the base
model itself, Qwen3.8-27B. At the time that
seemed like the right choice: it kept training stable, and
it kept the legal surface small, since distilling from
frontier models is typically not allowed under their terms
of use. It also gave us a data problem we did not fully
appreciate until 0.4 surfaced it, in the form of a model
that had learned to imitate itself a little too well.
To address it we built Deriva-0.4 on a much wider mix of
data. Synthetic data still made up about 70% of the corpus,
but it was no longer coming from one source. We pulled from
Qwen3.8-27B as the base, from frontier-grade
models including Kimi-K3 (hosted on our own
compute, locally, and sifted through carefully), and from
our own Hikaru-0.6, used specifically to shift
the personality away from Qwen's default.
Hikaru also let us strip out a layer of Chinese
state-aligned framing and push the model toward a more
Western edge, while keeping the raw capability that makes
Qwen useful in the first place.
Alongside Deriva-0.5 we made a deliberate cut: the OMEGA reasoning mode is out, at least for this stage of the project. OMEGA was always the most ambitious part of the design, a reasoning mode aimed at long-horizon, high-difficulty tasks, and it is too heavy for the model right now. Putting it back in while the base is still learning to size its own thinking just amplifies the overthinking problem instead of solving it. We plan to reintroduce OMEGA once Deriva has proved itself out, most likely somewhere around Deriva-1.1 to 1.2, possibly later.
Deriva-0.5 was meant to be the version that pulled everything together. It was not. In terms of personality, 0.5 produced little to no improvement over 0.4, which is a real disappointment after a build that was supposed to consolidate the gains. We are still investigating exactly why, and we will share what we find once we do. For now, Deriva-0.5 sits as a let down: a useful data point, not a milestone.
Deriva-0.6 is in development, and the early signs are more encouraging than anything we have seen since 0.2. The clearest improvement is in long-context understanding and alignment: the model stays in character, holds thread through much longer exchanges, and does not spiral into the overthinking loop that defined 0.5, even at the kind of turn depth where 0.5 would start to lose its footing around turn 70.
Our working theory is that this is a data effect, not a training effect. The 0.6 mix leans more heavily on high-quality examples in the 100-turn range, where 0.4 and 0.5 were comparatively thin. We are not ready to call 0.6 a win, but it is the first build in a few weeks that has felt like genuine progress, and we wanted to share that signal honestly rather than wait for a final number.
Development continues. We will share more as the next internal builds churn out, including what we learn from 0.6 once it finishes training, and whatever the 0.5 investigation turns up. For where Deriva-1 stands today, see the Deriva-1 model page and the earlier Update 1 and development posts.
