Comments - Olmo Hybrid and future LLM architectures
SubscribeSign in
Olmo Hybrid and future LLM architectures
The latest Olmo model and discussions at the frontier of open-source post training tools. Read →
4 Comments
Top first
Interconnects AI reply rules
Mar 5Edited
Liked by Nathan Lambert
This is FINALLY answering the question everyone has been thinking after reading the new model releases this year: why these hybrid attention layers? And apparently, more expressiveness is the answer?
I am more than excited to hear the post-training challenges for MoE models: if we can inference with sharding, couldn't we do the same with post-training as well?
Thank you for reading the article rather than text-to-speech. This has been more than refreshing.
Reply (1)
Author
my voice overs aren't perfect but the humanity is nice, and they help me proof read 🫡
Reply
Thank you for your time and attention concerning this recent discovery.
https://substack.com/@sublius/note/c-224332811
Reply
"Why does expressive power help with efficiency? This is where things are more nuanced. We argue that more expressive models will have better scaling laws, following the quantization model of neural scaling."
A fairly interesting paper you've linked. It's hard to tease out these at once endlessly expressive decoder models and at once not expressive in the ways we'd like
Reply