[EN] liam. - el fundador de una empresa china de IA valorada en más de $20,000,000...
Transcripción
URL17 de julio de 2026 · 39:17 min · 5440 palabras
Resumen automático
Open Models and Democratizing Intelligence
Open models provide full access to model weights, enabling deployment across diverse environments, including local servers and the cloud. This openness fosters transparency and accessibility, allowing users to utilize and improve models without being confined to proprietary "black box" solutions. Recent advancements have enabled open models to approach the performance frontier of proprietary models, making high-level AI capabilities more universally available. However, openness alone is insufficient; open models must also achieve state-of-the-art performance.
Three Key Dimensions of Model Scaling
Token Efficiency: Traditionally, scaling AI models involves increasing the number of training tokens, model parameters, and compute resources, which lowers loss and improves intelligence. However, improving token efficiency—achieving lower loss with the same number of tokens—raises the upper bound of achievable intelligence, especially as high-quality data becomes scarce. Enhanced token efficiency is not just about computational savings; it directly translates to better model performance.
Context Length: Increasing the context length allows models to handle more complex tasks by processing longer sequences of information. Transformers outperform LSTMs by maintaining lower loss over longer contexts, enabling applications such as codebase understanding and extended agent trajectories. Efficiently scaling context length is essential for modern AI tasks that require handling vast and complex inputs.
Agent Swarms: Moving beyond single-agent paradigms, orchestrating swarms of agents enables parallel execution of subtasks, increasing task capacity and reducing execution time for complex problems. This approach mimics organizational structures, where an orchestrator delegates tasks to specialized sub-agents, aggregates results, and iteratively refines outputs. Agent swarms can scale input processing, output generation, action-taking, and orchestration, making large-scale, complex tasks tractable.
Innovations in Token Efficiency: Muon Optimizer and QK Clip
Muon Optimizer: A second-order optimizer that transforms each gradient update to be orthogonal, differing from traditional Adam optimizers. Proper implementation of Muon can double token efficiency, effectively extracting more intelligence from the same data. Key techniques for scaling Muon include WeDK for large models and adjustable coefficients to maintain consistent RMS updates. A distributed implementation ensures memory efficiency across GPU clusters.
QK Clip: When scaling Muon to trillion-parameter models, training instability arises due to exploding logits. QK Clip addresses this by dynamically clipping the maximum values of query and key projections in each attention head, stabilizing training without affecting convergence. This technique enabled the successful training of trillion-parameter models with Muon.
Advances in Context Length: Kimi Linear and Delta Attention
Kimi Linear Architecture: Introduces Kimi delta attention, an improvement over previous gated delta rule (GDR) methods, by incorporating fine-grained, channel-wise decay factors (diagonal matrices) instead of global scalars. This allows selective retention or forgetting of information across channels, increasing model expressivity and efficiency for long contexts.
Efficient Implementation: The Kimi Linear approach uses mathematical reformulation to enable parallel computation on GPUs, maintaining efficiency without approximation. It outperforms both full attention and other linear attention variants on short, long input, and long output tasks, and is scalable to contexts of a million tokens or more.
Agent Swarms: Parallelism and Reinforcement Learning Objectives
Orchestration Paradigm: An orchestrator (main agent) spawns and manages sub-agents, assigns subtasks, collects results, and iteratively refines outputs. This structure enables parallel execution, reducing time and increasing effectiveness for complex tasks.
Scaling Dimensions: Agent swarms can process vast inputs, generate extensive outputs, perform actions at scale, and manage orchestration efficiently.
Reinforcement Learning Objectives:
- Instantiation Reward: Encourages the creation of sub-agents to promote parallelism and prevent defaulting to serial execution.
- Finished Reward: Incentivizes completion of subtasks, ensuring meaningful parallelization.
- Outcome Reward: Measures overall task completion, guiding the system towards effective solutions.
- Reward weights are decayed over training to balance learning dynamics.
Kimi K2.5: Integration and Emergent Capabilities
Unified Vision-Text Training: Kimi K2.5 is the first open model to natively fuse vision and text training from the outset (early fusion), rather than adding vision capabilities post hoc. This alignment in a shared embedding space enables advanced multimodal capabilities, such as vision-to-code and style transfer.
Mutual Enhancement: Vision training improves text performance, and a strong text base enhances vision capabilities, even without dedicated vision SFT data. Joint reinforcement learning over both modalities yields near state-of-the-art results in vision tasks.
Stable Large-Scale Training: The combination of Muon optimizer and QK Clip enables stable training over tens of trillions of tokens, producing robust base models for further fine-tuning.
Attention Residual: Next-Generation Depth Scaling
Concept: Inspired by residual connections (ResNet) and LSTM analogies, Attention Residual applies attention mechanisms across the depth (layers) of neural networks, not just temporally. Instead of only using the previous layer’s output, the model aggregates all previous hidden states via attention to compute the current state.
Block Attention Residual: To manage memory and communication overhead, layers are grouped into blocks; attention residual is applied at the block level, while standard residuals are used within blocks. This approach maintains efficiency with minimal accuracy loss.
Performance Gains: Attention Residual improves token efficiency by 24% and achieves lower validation loss and superior results on coding, math, and reasoning benchmarks.
Research Methodology and Outlook
The current era enables rigorous, large-scale experimentation and benchmarking, facilitating confident conclusions and rapid progress in model architecture and optimization.
Continuous innovation in open models, optimizers, attention mechanisms, and agent paradigms is expected to drive further advances in AI capabilities.
The open-source community is positioned to accelerate these developments, democratizing access to cutting-edge intelligence.
Summary
Key advances include improved token efficiency (Muon optimizer, QK Clip), scalable context length (Kimi Linear), agent swarm orchestration, unified multimodal training (Kimi K2.5), and novel depth-wise attention mechanisms (Attention Residual). These innovations collectively push the boundaries of open models, making high-performance, accessible AI a reality and setting the stage for ongoing progress in the field.
Open models provide full access to model weights, enabling deployment across diverse environments, including local servers and the cloud. This openness fosters transparency and accessibility, allowing users to utilize and improve models without being confined to proprietary "black box" solutions. Recent advancements have enabled open models to approach the performance frontier of proprietary models, making high-level AI capabilities more universally available. However, openness alone is insufficient; open models must also achieve state-of-the-art performance.
Three Key Dimensions of Model Scaling
Innovations in Token Efficiency: Muon Optimizer and QK Clip
Advances in Context Length: Kimi Linear and Delta Attention
Agent Swarms: Parallelism and Reinforcement Learning Objectives
- Instantiation Reward: Encourages the creation of sub-agents to promote parallelism and prevent defaulting to serial execution.
- Finished Reward: Incentivizes completion of subtasks, ensuring meaningful parallelization.
- Outcome Reward: Measures overall task completion, guiding the system towards effective solutions.
- Reward weights are decayed over training to balance learning dynamics.
Kimi K2.5: Integration and Emergent Capabilities
Attention Residual: Next-Generation Depth Scaling
Research Methodology and Outlook
Summary
Key advances include improved token efficiency (Muon optimizer, QK Clip), scalable context length (Kimi Linear), agent swarm orchestration, unified multimodal training (Kimi K2.5), and novel depth-wise attention mechanisms (Attention Residual). These innovations collectively push the boundaries of open models, making high-performance, accessible AI a reality and setting the stage for ongoing progress in the field.
[00:00] Hi, everyone. Thank you so much for the introduction. It's great to be here. It's great to have this opportunity to share with you guys some of our latest progress and efforts. So, one of our major pursues, is to build better open models, and we believe in democratizing intelligence.
[00:24] With open models, you can deploy anywhere. It can be on your local servers. It can be on the cloud, and you can access every single bit of the weights in the model instead of just using a black box. And this is one of the slides that I took from Jensen's talk earlier this year at CES. So as you can see, open models are quickly closing the gap with proprietary models and is reaching the frontier.
[00:52] And we believe that with better embedded open models, we're going to make intelligence more accessible to anybody in the world, in every corner of the world. But open models cannot be just open. They have have also to be great. So in this talk, we're going to discuss how we make open models great. So as we know, scaling is a primary driver for a lot of progress, maybe all of the major AI developments that we have witnessed in the last few years.
[01:28] And here we're going to discuss how we scale our model in different dimensions. So on the left hand side, the first figure you see here is kind of the the standard scaling law. So on the x axis, you have the log of the number of training tokens, and on the y axis, you have the log loss. And as you scale the number of training tokens, you get a lower loss. But here, the point is we're not going to just scale the number of training tokens, but we also want to improve the token efficiency, meaning that we want to move this curve to the left hand side so that we can achieve a lower loss, a much lower loss using the same number of training tokens.
[02:12] And this can be achieved by having better architectures and optimizers as discuss in our later slides. And the second scaling dimensions that we're very interested in is to scale the context length. So as you can see in the second second figure, if we increase the context length, then we can, have a much higher accuracy in terms of predicting the token loss at a given position. And this means that we can increase the capability of the model to achieve more complex tasks by increasing the context dense. So this is the second scaling dimensions that we're going to talk about.
[02:55] And the third scaling dimension is the number of agents. So we introduced this new learning paradigm of agent swarms where we can, we don't just rely on a single agent, but we also orchestrate a swarm of agents that can accomplish the subtask in parallel so that we can increase the task capacity. And we can translate all of this into the language of agents. If you look at token efficiency, it's mostly about having a stronger prior so that you can have more efficiency when you do agent IRL to search for a better solution. When think about long contacts, it's mostly about increasing the contact length so that you can have a longer rounding agent.
[03:39] It can probably run for days or even weeks or months to accomplish more tasks, more complex tasks. And about, and for agent swarms, it's another dimension that add to it. And at the end of the we're going to have a swarm of agents that each of them have a super long context, and each of them have a very strong prior for us to search in this entire agent IRL system. All right, so we're gonna start from token efficiency. So this is one of the most classical figures in the history of machine learning.
[04:17] So it's taken from Kaplan et al. And it basically says that if we scale proportionately the number of training tokens, the model parameters, and also the amount of compute, we can get lower and lower loss. And this is, you know, one of the major breakthroughs that the entire community has achieved in the last few years to get better intelligence. But here, what we're interested in is to have better and better token efficiency. Here's the thing.
[04:49] So one thing that I would like to emphasize is that token efficiency is not just about efficiency. It's actually also about improving the upper bound of intelligence. So here's here's here's why. So, suppose you have, say, trillion tokens, 50,000,000,000,000 high quality tokens, and then you apply this new optimizer, maybe the Muun optimizer, and then all of a sudden you have a two times token efficiency. So it means that it's almost like magic that you get equivalently 100,000,000,000,000.
[05:26] And nowadays, we are scaling towards the data war and we're hitting, you know, the data war. And the amount of high quality data is quite limited. And if we suppose that is a constant amount, then we increase the token efficiency. It means that we're going to get better intelligence out of it. It's not just about infrastructure efficiency.
[05:47] It's about, you know, better intelligence. So so this is why we spend, you know, a lot of efforts in this aspect because it's going to push the frontier of of intelligence. And Muon optimizer is one of the things that we have heavily invested in, since last year. So it's a second order optimizer, and basically every single gradient update is transformed in a way that each entry is orthogonal to each other. And this is very different from the traditional Adam optimizer.
[06:23] And if if we implement this optimizer properly, you can get a two times token efficient efficiency improvement. So we we are one we are the first work. We published the first work to demonstrate that Muir Own optimizer is actually scalable for LLM training. And these are two key techniques that we employ to make it effective for large scale training. So one of them is WeDK.
[06:49] It is critical for scaling to larger models. And the second is we want to ensure a consistent RMS updates compared to Adam. So we have this adjustable coefficient that is applied to each update so that the resulting RMS is going to be comparable to Adam. And to make MIOM memory efficient across all these NVIDIA GPU clusters, We also develop a distributed optimizer implementation that partitions the space across the data parallel group so that we can have a very efficient implementation for the MIUM optimizer. And these are some of the results that were presented in the paper.
[07:35] So as you can see, with the same number of parameters and the same number of training tokens, we just replace the original add and w optimizer with the new Muon optimizer is going to to improve the performance across the board, significantly. But there was, this new challenge that we encountered when we tried to scale it up further. When we try to scale Muon for a 1,000,000,000,000 perimeter model, we encounter a new issue about training instability. So as you can see on the left figure, we observe that the MAX logits quickly explodes and quickly exceeds 1,000. And the typical values for training for this max logits is about, say, 50 or maybe less than 100.
[08:27] But for meal, it quickly exists 1,000. And at the same time, we observe training divergence on the left hand side. If you look at the training loss, it goes down a bit, but then at the end of the day, it exposed and it cannot converge as expected. So this is one of the technical challenges that we have to adjust. And the solution to this is to introduce this new technique called QK clip.
[08:54] So basically, what it says is that for each attention head in this entire neural network, we're going to in the forward path, we're going to compute the max logic, and then we're going to calculate a dividing factor that can be applied to each key projection as well as the query projection so that we can sort of clip the maximum value of the query and the key to sort of constrain it into a given range so that we're not going to have explosion anymore. So these are some of the empirical results. On the left hand side, there are two curves, but they are strictly overlapped with each other. So these are the training curves before and after applying the clipping technique. So you can see the clipping technique does not affect the training loss decrease at all.
[09:48] But on the right hand side, if we inspect the intermediate metric, if we inspect the max logic, it's going to be effectively constrained. So first, expose as before, but at the value of 100, it's going to be clipped at a constant value for a long time. And then after a certain number of steps, will just naturally go down. So, the neural network sort of find a way to constrain the maximum value of the max logic to ensure a stable training process. And at the same time, it doesn't affect, you know, the training convergence as shown in left figure.
[10:28] So we employed this technique in our K2 model training and successfully scaled it to 1,000,000,000,000 parameters. And this is the first example of a large scale muon training in the history of machine learning. And the second dimension that we're very interested in is lung contacts. So this is another figure. It's probably last known.
[10:55] It's it's one of the hidden gems in these papers. So instead of just, you know, pushing down the training laws by training on more tokens, it has some insights from another perspective. So as we can see, this is a comparison between transformers and LSTMs. So on the left hand side, we can see that transformers achieve a lower training loss given the same number of parameters and the same number of training tokens as expected. And this is why transformers become, you know, sort of the de facto architecture that people are using right now.
[11:30] But on the right hand side, it's really interesting to see that transformers are actually better because it can improve through the whole context. So the x axis is the token index in context. If you increase the token index, you can see that the training loss of transformers actually dropped by a lot. If you just continually increase the context lens, the loss just continuously drops down. But if you look at the curve of RSTM, it's saturated after a certain number of tokens.
[12:04] It means that transformers have this better capability of capturing longer context, And this what makes it better because if it go back to like ten years ago, people use LSTM for tasks like machine translation, but it is not good for, for example, understanding entire codebase or running a super long agent trajectory to solve, for example, Linux kernels from scratch. It's not going to be accomplishable by STMs. So this is a very much needed capability in the era of agents because tasks are becoming harder and harder, and we need longer and longer contacts. So the research idea here is to develop a better architecture so that we can efficiently scale to a longer contact lens and at the same time achieve a lower per token loss at larger token indices. And this is the motivation for which we introduce this new architecture called Linea.
[13:11] And it contains this new linear attention variant called Kimi delta attention, which improves the original gated delta rule, GDR, by improved recurrent memory. I will show the details later. And at the same time, we're going to mix linear attention layers with full attention layers using a one to three ratio so that you can balance between these long context capabilities and, at the same time, having a more efficient implementation. So this is some of the formulation. The basic idea is simple.
[13:51] If you look at linear attention, in the original formulation, the memory is going to be global. So there is a global single decay factor that is applied along the way. So it means that, basically, if there are only two cases. In one case, you're going to forget basically everything, and you're not going to retain any information. And in the second case, you can choose to retain, you know, almost everything, but at the same time, you you don't have the capability to leave out some of the unnecessary information in this long context.
[14:26] So we introduced this key idea of having a fine grained decay factor as shown in this highlighted alpha term. So it's going to, instead of being a scalar, it's going to be a diagonal matrix, which controls the decay rates for each channel so that we can have two possibilities. For some of the channels, we can decay really, really slow, meaning that we can return this long context information across a very long range. And at the same time, for the other channels, we can quickly forget the information from the past indices to refresh it and observe new information. And this is to increase the expressivity of this model.
[15:15] And of course, to leverage modern GPUs, we have to use this Chang Kwei's formulation so that we can paralyze the computation on modern GPUs. So the first equation here is the Chang Huize formulation of kimi linear. But as you can see, this is going to bring massive infrastructure challenges because of this newly introduced alpha term. Because now it is the metrics instead of a scalar, it cannot easily be factored out. So to achieve an efficient implementation, we rewrite the entire equation into the bottom three equations.
[15:56] So we introduce this matrix inversion operation, as well as introducing the cumulative decay factor so that we can implement this entire thing in parallel without sacrificing any efficiency. And more importantly, this is not an approximation. It's an exact mathematically equivalent formulation so that we can achieve much efficient implementation without sacrificing any loss in terms of performance. So it's going to be as efficient as, you know, previous linear tension variance, but at the same time, much more expressive. So these are some of the results that we obtained using a fair comparison.
[16:43] So on the left hand side, we see the performance on two different types of tasks. So MMAU is a short context task. So for short context task, Kimi Linear achieved a better performance compared to MLA and GDM. And at the same time, for longer context tasks such as ruler, kimi linear is also better than the other variants while being much more efficient compared to MLA. And when we scale the contact lens further to, for example, 1,000,000 tokens or even longer, it's going to be much more efficient compared to the baselines.
[17:24] And this is also the first architecture that can outperform full attention across the board, including short context tasks, long input tasks, and long output tasks. So these are two key dimensions that we are interested in. And the third dimension is the agent swarms. So here is a diagram to showcase how we design this agent swarm paradigm to solve some of the more complex tasks compared to single agent paradigms. So here, we have an orchestrator, or you can call it a main agent.
[18:04] It's responsible for orchestrating tasks. It has different options. For example, we can spawn a group of sub agents and assign new tasks to these sub agents. Or you can collect the results from the return of these sub agents, and you can sort of performing this process in an iterative way. And at the end of day, you can accomplish a more complex task compared to using one single agent.
[18:32] And it's analogous to human society. For example, if we build a company, we need different roles, and we need, for example, orchestrator or maybe we need a CEO to decompose and assign the tasks to different roles. And then at the other day, the entire organization is going to have to move towards this same goal. And here, for example, in this case, we have maybe you have the AI researchers, you have the web developers, you have physic researchers, and they can study different topics. And at the end of the day, you just collect the results and spawn a group of fact checkers and web developers and file downloaders to assemble the results into a single report.
[19:20] And this is another perspective to look at this new paradigm. So the x axis is the complexity of the task, and the y axis is the execution time. The complexity of the task is measured by the accuracy of a group of models on such tasks. We can see with agent swarm, it's going to substantially reduce the execution time compared to single agents. It's going to be more effective.
[19:58] And this means that we can scale this agent's ROM paradigm to, for example, if you run this agent's ROMs with 100 or maybe even 1,000 agents, you can accomplish a complex task within a certain period of time that is tolerable for producing real economical value. And we can certainly scale it in different dimensions. We can scale the inputs. For example, we can download and read hundreds of sources or even maybe thousands of doses in parallel. Or we can outputs, write a 100 page literature review in parallel.
[20:39] We can take actions at scale. We can perform data analysis for 10 different tasks. And also, it is orchestration at scale. You have to learn to design subtasks and aggregate the results. And technically, we define some new objective functions to guide the learning process of our agents' realm system.
[21:02] So there are three real functions, real objectives that are considered here compared to the conventional single agent IO learning. So the first term is what we call the instantiation reward. It incentivizes sub agent instantiation to prevent this serial collapse phenomenon from happening. So basically, we don't want it to default to single agent execution. We want to encourage the parallel executions, especially when it's early stage in training.
[21:45] And of course, we can decay the weight for this instantiation reward time over training course because when it learns parallel execution, we can reduce the wait. And the second term here is finished reward. And it is used because we observe one of the things in training that some of these subtasks are just created but never finished. So it's almost like it's going to hack the first time by just spawning a bunch of sub agents and the task might be too complex or maybe the task just doesn't make sense. And here, we use this finished reward to basically encourage that each of the subtasks should have a relatively high ratio of completion.
[22:34] Instead of just spawning a bunch of pseudo tasks, we need it to be meaningful. So this is the second term that we use. And of course, we use the same decay strategy. We use a relative highway at the beginning of training, and we decay it to a relatively low weight at the end of training. And of course, the third term is the standard term.
[22:55] It's the outcome reward. It's going to measure whether the entire task is completed. And then we're going to add these three terms in our reinforcement learning system. And of course, we have to build, you know, the entire infrastructure because right now you need to support the parallel execution and then you'll need to support different reward functions and to, you know, maximize the efficiency of the entire agent's wrong IO system. So here are three different things that we have tried scaling.
[23:32] The MuonClip optimizer improves token efficiency, and KidMe delta attention in the Kimi linear architecture improves long contacts. And we also have the agent swarms paradigm to further create a new dimension of scaling. And all of this put together, we created Kimi K2.5, a new model that we just released over one month ago. Here's a short video to demonstrate some of its capabilities. So, yeah, there are a lot of interesting things, capabilities that we discover from the model.
[25:13] For example, it merges the visual capabilities with coding capabilities. So a lot of new things just emerge out of it. It can read a video and then produce a website that sort of replicates or style transfer the original video. And all of these are due to successful and stable training at the pre training stage. So this is also one of the most beautiful curves that I observed in my life.
[25:44] So this is the training curve of the K2.5 based model. So as you can see, it went through over 15,000,000,000,000 tokens, and of course, in K2.5, we additionally trained another 15,000,000,000,000 tokens. And the entire training process is just so stable. There's no lost spike, especially when we introduced this new Muon optimizer. We didn't observe any spike.
[26:09] And this smooth, stable training process produces a very stable outcome, a very strong base model that we can fine tune on top of it to achieve new capabilities as we introduce and saw in the video. And this is also, of course, trend on NVIDIA h 800 GPUs. And you should know in this h 800 cluster contains two TB RAM and eight GPUs. They are connected by NVIDIA. And one of the another, you know, key innovation of KiMeK 8.5 is that it is the first open model with native joint vision text capabilities.
[26:52] So if you look at previous open models, usually, their visual capabilities are added on top of a text base, meaning that, for example, if you train the text models for 20,000,000,000,000 tokens, and then on top of it, you do another 2,000,000,000,000 sort of a post training process to add additional visual capabilities on top of it. But for K-two 55, it's different in the sense that we fuse the training process of vision and text from day one. So it's called early fusion here. We start from, you know, 0% of the progress. So from day one, we're going to merge the vision and text tokens, as shown in preliminary experiments, it outperforms late fusion.
[27:37] And some of the new capabilities that we observe also come from this training recipe. For example, if you want to do vision to kill, you really have to merge vision and text into a single brand to achieve that. If you separate these two brands, it's not going to happen. You have to align these two modalities into a shared embedding space, a shared representation space so as to achieve this. And another interesting thing that we observe is that these two modalities can actually enhance each other.
[28:13] So that's been long been a challenge that if you add vision capabilities into a text model, it's going to somewhat hurt the text performance. But here we found that if you train it properly, these two modalities can actually enhance each other. So this is one of the key findings that we observe in in our training. So first, vision improves text. So this is so interesting.
[28:40] So before vision arrow, the performance in the first column, and then we have the performance after vision arrow. So here, vision arrow refers to a process that we only use vision tasks. So there is no test task involved here. We only have vision tasks. For example, we teach the model how to how to count, how to answer some of these visual QA problems without any, for example, math, any coding problems in in in this space.
[29:10] But we observe that it's going to improve the performance for even, you know, reasonably heavy text task. And on the other hand, text also improves vision. If you have a very strong text base, you you actually don't need any vision SFT data in the training process. And this is the approach that we adopt. So it's called zero vision SFT.
[29:31] Basically, we don't have we have basically zero vision SFT data. And the only SFT data that we have is the text SFT data. And then we do a joint IR over text and vision. And you can see that we can achieve almost state of the art performance across the board on vision tasks without any vision data. So it's clear that if you have a strong text base, it's also going to improve the vision if you align these two modalities into a shared space in your pro training.
[30:05] And also, these are some of the examples of yeah. As I was showing in the video, so it's it demonstrates strong capabilities of visual design and front end coding, and this also emerges from our vision text through training. So, after all this, so this is all about Kimik 2.5. And as probably, you probably know, we released our new architecture yesterday in our tech report. It's called Attention Residual.
[30:42] So here I'm also gonna briefly talk about our new work, which serves as a sneak peek into our next generation architecture that we're probably going to adopt in our later models. So here the motivation is quite simple. Can we apply some of our techniques that we use in the temporal dimension and we just take some of the inspirations and then apply it to the depth dimension. So and it starts from this residual connection. So I still remember listening to to Camming's talk at a tutorial in in ICML twenty sixteen, ten years ago.
[31:26] So it was a brilliant idea. So basically, before ResNet, nobody was able to train deep networks. If you increase the depth, if you increase the number of layers for neural networks, nobody was able to train it because you observe this gradient exposure, gradient vanishing, all these, you know, stability issues. But then after the introduction of ResNet, we can train, you know, an arbitrarily large number of layers. You can stack as many layers as you want, and you don't have to worry about the training instability issue and stuff.
[32:02] And as discussed in Ilya's talk two years ago, it basically says that residual connection is a variant of LSTM, but just rotated 90 degrees. So how do you understand this? If you look at LSTM, it's a variant of recurrent net, right? And it's a recurrent model process. So we're going to take the hidden states from the last step, and then we're going to have some gating mechanism, some function to produce the current states.
[32:35] Right? And if you look at the depth dimension, the residual connection is basically the same. We're going to take the outputs from the last layer, and then we're going to apply some sort of function on top of it to produce the current outputs of the current layer. It's just the formulation is different. For example, for a residual connection, we're going to use a fixed addition.
[32:59] We're going to have this, we're going to add the previous hidden states with the current output. It's just a formulation that's different, but the basic idea is the same. It's a recurrent net applied in the dimension of depth. And but on the other hand, we can think about reformulating this this function. Instead of having an STM, can we have an attention in the dimension of that?
[33:29] And it's going to create new possibilities because attention have has been demonstrated to be so successful, in the transformer era. So what we're gonna do is not just to take the last hidden state, but we're going to consider all the previous hidden states and use the attention operation, the attention mechanism to assemble and aggregate all of these previous hidden states to compute the current state. So this is exactly a tension rotated by 90 degrees. It's sort of we view it as a natural generalization of residual connections in the LSTM analogy. Okay.
[34:09] And here is the detailed formulation. So on the left hand side is a standard residual. As I said, it's basically LSTM rotated by 90 degrees. And the second figure is attention rotated by 90 degrees. What we do is to collect all the previous hidden states and have a simple attention operation on top of it to produce the current layers outcome.
[34:33] Of course, increase the efficiency, to reduce the infrastructure, for example, communication and memory overhead, we also designed a new variant called block attention residual on the right hand side. So basically, the idea is also simple. We're going to divide all the layers in the neural networks into multiple blocks. For example, each block can contain, say, 16 layers or it can contain maybe four layers. And then for each block, we're going to apply this attention residue only on the outputs of each block.
[35:09] But within each block, we still adopt this standard residue. So this is going to reduce a lot of overhead while having minimal loss in terms of training accuracy. And these are some of the, you know, impressive results that we achieve on this new architecture. So on the scaling law, we can improve the token efficiency by 24%, meaning that if you have 50,000,000,000,000 high quality tokens, now you just magically have over 60,000,000,000,000 tokens. And then for the validation loss, you can also observe that it's consistently lower than the original curve, demonstrating the stability across optimization, and also achieve the best improvement on some of these coding, math, and reasoning heavy tasks as shown in the benchmark results of GPQA, math, and human eval.
[36:12] So the entire community keeps moving forward, and we're happy that we can we're able to contribute to to the community with, you know, new technologies. And some of these technologies have been sort of standard and de facto for a long time. But as you can see, we still see a lot of opportunities to improve it, to have revolutionary new design to achieve better performance. If we multiply all these scans together, you can actually have a much better model. So, Adam was invented in 2014, and now we scale an open source MuonClip, a dropping replacement for Adam.
[36:57] And I'm sure that if you're training transformer LLM, it's going to be much better if you use MuonClip instead of Adam. And attention was invented over eight years ago, and then now we have chemilinear, which is a linear version. We don't have to use full attention across all layers. We can have linear attention that performs better on short, long contacts at the same time. And also residual connections are now also challenged.
[37:28] We scaled an open source attention residual. So I think one of the interesting things about our era is that we sort of adopt a different mindset doing research. So if we go back to ten years ago, it's mostly about publishing a new idea. But then I think the lack of the rigor of the experiments is very hard to produce solid experimental results. But now we have the scaling ladder.
[37:58] We have enough resources to, you know, trend the model and run it on different at different scales. We can have, you know, a whole set of benchmarks to measure the progress. So it is it becomes easier to make confident and solid conclusion out of it. And this is one of the reasons why we are observing, you know, new progress on this ancient techniques. And I'm sure that we'll see more and more, especially in the open source community.
[38:28] I think we're going to have more and more, even better architectural and, you know, optimization improvement in in the next few years. Alright. So to summarize, we're going to keep scaling our models. And so these are three dimensions. For example, we see different architectures and optimizers that optimize all three dimensions, and we'll keep seeing new dimensions for scaling.
[38:59] Agents' ROMs is not the end, and we are glad that we can move forward with the entire open source community to achieve better and better intelligence. Thank you so much.
Empieza a transcribir gratis
Desde YouTube, WhatsApp o graba directo desde el navegador. Resúmenes, traducciones y preguntas con IA incluidas.
🎥
YouTube & TikTok
📱
Bot de WhatsApp
🎙️
Graba desde el navegador
✓ Plan gratuito disponible siempre · Miles de usuarios confían en TranscribeGo