The Bottleneck to the Bottleneck: A Study Analyzing RLVR Inputs

July 5th, 2026

Zachary Speck

Published environments: vc-deal-math · vc-deal-extraction


What is going on in our world?

I think society has amnesia. Amnesia about our previous world, not just from a decade ago, but even from 5 years ago. Everything used to move so slow. However, with the introduction of high-velocity artificial intelligence, it seems that anything within the realm of scientific research is possible now within a compressed timeline. Even the study which I will be highlighting in this piece wouldn’t have been possible to run in a month without the assistance of AI, and alternatively I likely would’ve taken 3-6 months to hand-code the scaffolding and architecture. In a world where AI is in seemingly every conversation I participate in (and eavesdrop on), there are of course naysayers. These naysayers belabor on about how AI is stripping away our humanity and taking jobs and stealing water and disconnecting us from reality, and, and, and… I am an AI optimist in the face of these claims. It is not the pessimists’ fault for taking this view—they simply just don’t have full context. To help you understand where I am coming from, let’s think about the most remarkable scientific achievements in recent years:

What is the common denominator here? AI. AI was either used directly in the discovery, or as the most meaningful accelerant. Zero of these accomplishments would have been possible a decade ago, and herein lies why my optimism about the technological pageant of the future is unwavering in the face of shameless critics. Let’s think about the future and the most pressing problems of our generation including but not limited to: climate change, overpopulation, hunger, curing cancer, and even the meaning of life. With every single one of these problems, I can undoubtedly guarantee you that AI will be the most significant contribution to accelerating the solution. I would go as far to say that ignoring this may actually be selfish. I digress, my intellectual warfare with skeptics is not what I am writing about.

What I really wanted to put under a microscope is the underlying engineering process driving AI today: Reinforcement Learning (RL). Upon researching this industry, I noticed a problem that no one else had researched, and decided to dedicate a month of my life to researching this problem at a small scale. The results I found were conclusive, and can hopefully act as a meaningful data point on understanding how we can accelerate AI progress to its maximum threshold. Before detailing my study, it’s important to get a lay of the land in case you are unfamiliar with RL entirely.

An instrumental master of puppets

In this unbelievably hasty AI world, a plethora of subindustries solely focused on model training have emerged. Most notably, RL has increasingly become the primary method of post-training that the frontier AI labs tweak their models on. As defined by AWS, RL is a machine learning (ML) technique that trains software to make decisions to achieve the most optimal results. It mimics the trial-and-error learning process that humans use to achieve their goals. Software actions that work towards your goal are reinforced, while actions that detract from the goal are ignored. Let’s zoom out for a second and think about AI training in general.

The lay of the land. This diagram shows machine learning's subdivisions and where RLVR sits.
machine learningsupervisedlearningunsupervisedlearningreinforcementlearning (RL)how a frontier model gets trainedpre-trainingpost-trainingSFTRLHF / RLAIFRLVRRL environmentcompanies
The top panel shows the three classic subdivisions of machine learning. The bottom panel shows the frontier-model training pipeline. Pre-training comes first, then the post-training chain of SFT, then RLHF or RLAIF, then RLVR. RL environment companies supply the environments RLVR trains in.

There are two halves of frontier model training. Pre-training and post-training. Pre-training exposes the model to the entire internet and tries to get the agent to predict the next sequence of text in an existing sentence. Over the past decade, this is the methodology that has contributed to the majority of LLM progress. However, as Open-AI and Thinking Machines co-founder John Schulman has noted in recent times, pre-training has quite clearly hit diminishing returns. Understanding why is simple. There is only so much existing stuff you can teach a model. To tackle the gargantuan (and neverending) problems of our world today, you need a model that can genuinely learn new skills.

Hence, welcome to the world of post-training. The first major leg of post-training that we have seen is data-labeling, with the emergence of companies like Scale AI and Mercor. Data labeling is the process of identifying raw data (such as images, text, or video) and adding informative tags or annotations to give it context. This creates the foundation for supervised ML, allowing models to recognize patterns, learn correct answers, and make accurate predictions on new data. Labeling used to be performed by normal people, but given the intellectual limits of humans, these days it is primarily outsourced to individuals with very high intellectual capacity and specialization, such as PhDs and professional scientists. But even now, it is very clear data labeling at its most effective will plateau. Think about it this way—Dario Amodei thinks that AI will be smarter than a Nobel Prize winner in the next 12-24 months. If this is the case, there is zero benefit to humans orchestrating the data labeling process, since the model’s intelligence will be capped at the most intelligent human’s brainpower. Humans simply just aren’t smart enough, which may be uncomfortable to hear. You should embrace it though. To solve the problems of the future, we need to outsource intelligence that is beyond our capabilities. So this leaves only one leg of post-training left: RL.

RL is not really a new concept—it has existed in psychological research for nearly a century, and is only now being applied to AI training. To make RL a bit more clear let’s take a simple example: training a dog. You give the dog a circumstance, and if the dog acts in the way that you wanted it to (grounded answer), then you give the dog a treat (reward). Now, the dog has been reinforced for that behavior and is more likely to do that behavior instead. It works the other way around too—if the dog does something that was incorrect then it is reinforced to not do that behavior again.

There are 3 primary forms of RL. RL from humans (RLHF), RL from AI (RLAIF), and RL from verifiable reward (RLVR). I will solely focus on RLVR in this piece. RLVR usually ends up being the last natural sequence in the post-training chain, given that it involves an auto-grader grading from a standardized correct answer. RLHF and RLAIF end up hitting natural plateaus themselves, with RLHF being capped by the limitations of human intelligence and RLAIF being jammed from AI answers being too subjective. RLVR however, may be the key to teaching frontier models genuinely new concepts.

Before praising RLVR as the holy grail, it’s important to understand the current debate. There has been some pushback from professional researchers on the validity of RLVR. It is important to point this out before detailing what my study aimed to accomplish since I assume from the outset that RLVR is effective. Skeptics like Andrej Karpathy claim that RLVR just surfaces existing knowledge rather than actually training new knowledge, and that “outcome rewards are like sucking supervision through a straw.” In other words, he is saying that the inputs required for RLVR are not worth the incremental learning benefit that you receive.

Whether you believe in RLVR or not, you cannot ignore its instrumental presence to RL. If you are an AI optimist like me, then you probably also believe there to be a genuinely better world on the other side of AGI. With that world in the imminent future, the pace at which we break through new plateaus becomes increasingly critical, and understanding which inputs have the highest leverage in a post-training world helps us understand how we can move towards this utopia. In this regard, I view RL as the master of puppets in an AI-driven world.

Putting the RL chain under a magnifying glass

Before I dive into my experiment, I want to map out the different flavors of RL. Surprisingly, this market is ridden with niches that each capture different pieces of the value chain. I want to analyze each one of them to highlight their far-reaching capabilities.

Data Labeling. Let’s start by observing the giant data foundries. Classic pre-training. Scale AI, Surge AI, Mercor, Turing, Micro1, Handshake—the list goes on and on, but these are the data labeling giants that dominate pre-training. As I previously touched on, these companies train AI models via human-generated training material in a process called Supervised Fine-Tuning (SFT). SFT is showing the model the question and then the answer, making the model memorize both and attributing each to each other. This rather simple method of training has worked for a while, but only for a while can it continue to provide requisite returns. Because of this diminishing returns curve, every single data labeling giant has since pivoted to RL. Scale AI launched dedicated RL environments in early 2026, Surge AI has spun up an internal environments division, Mercor has started pitching environments, and Turing has also started building UI gyms and RL environments. RL is where the world is going, and there is no denying it. Whether the labeling giants will survive in this post-pre-training world is beyond the scope of this piece, but I will note this is a fiercely competitive environment.

RL Fabs. With RL infrastructure becoming a bottleneck in AI, the leading labs like Anthropic and OpenAI can’t generate the requisite RL supply necessary to continually train models, leading them to outsource RL capabilities. This is where the RL pure-play market has exploded—most notably Mechanize and Prime Intellect. These companies sell entire RL environments optimized to accelerate the pace of learning and progress towards AGI. Mechanize operates closed-source and sells a very small number of very robust environments heavily focused on coding and SWE, and their mission statement is to automate all jobs. May sound scary to some. Prime Intellect on the other hand is known as the "GitHub for RL environments”—they are completely open source, and it is where I actually posted the study that I have outlined in this experiment. The rest of this menu is purely seed-stage—companies housing <20 employees, with 1-3 lab customers at most, and mainly under exclusive closed-source contracts. The most notable companies on this roster are Habitat, DeepTune, Fleet, Vmax, Preference Model, Bespoke Labs, and Veris. While these smaller pure-plays haven’t achieved anything publicly extraordinary yet, who knows where they will be in a decade? 3 years ago you didn’t know what Anthropic was. Therefore, it may very well be that in 3 years one of these RL fabs may end up taking the AI throne. There is a bear case to be made on pure-play RL fabs with labs insourcing this problem themselves. xAI has direct “AI tutors” rather than leaning on vendors, and DeepMind has also started an automated materials-science lab. There is no telling whether these case studies are the canary in the coal mine, but with Anthropic planning to spend $1B on RL environments, this industry isn’t going anywhere soon.

Evals. The next layer in the RL stack is eval infrastructure. Eval companies’ entire job is to be the grader outside of the training loop that observes training and catches regressions before applications on top of the model are built. Some notable eval pure-plays include Braintrust and LMArena, with some of the data labeling giants building their own eval infrastructure in-house, like Scale with SEAL, or Mercor’s research arm. Instead of selling to labs, these evals are mainly selling to product teams who build applications on top of leading models, and their entire purpose is to be the outside “gradebook” that catches mistakes and ensures model performance is on-par with what is being shipped to a live customer. Although this likely has the thinnest TAM in the RL stack due to budgets being drained on RL environments, this does not mean in the slightest that it’s the least important.

Physical RL. Another subindustry critical RL space is RL for science. These are companies pushing environments into the physical world for grounded rewards. The hottest startups in this arena are Periodic Labs, Medra, FutureHouse, and Isomorphic Labs. These companies own actual labs and run real wet-lab scientific experiments to collect ground truth and data to package up and sell. They do this because modern scientific research tends to disagree quite frequently, and in a world where AI will be instrumental in scientific breakthroughs like drug discovery and new materials, they are manufacturing a single source of truth grounded in reality. In order for RLVR to work, the grader has to be immutably correct. And in these experiments where there is a physical and grounded truth, it may be the most valuable data you could hope for given there is rarely a common ground in the scientific research community.

RLaaS. The final relevant RL vertical I want to cover is RL as a Service (RLaaS). These are the companies that just do RL for you. Think of a gym. With the RL pure-plays, the leading labs are the ones that own the gym memberships and companies like Prime Intellect and Mechanize are just giving the gyms for Anthropic and OpenAI to workout in. Whereas with RLaaS, companies like Thinking Machines and Applied Compute are like personal trainers that actually train the labs themselves and just run the RL process.

A probable future is that out of the ~20 RL hot-shot companies, intense consolidation will likely occur with maybe 3-5 surviving as a base case as humanity moves AI onto the next leg of post-training. Nevertheless, I believe it is critical to understand and study the methodologies in which these entities operate to deeply understand how technological progress is being made.

The unresearched gap keeping me up at night

A couple months ago, I started reading into this industry for the first time, and was absolutely fascinated by RLVR. Out of natural curiosity, I went on arXiv and scraped every single RL-labeled study that I could find in May to mid-June of 2026, and upon reading it became very clear where AI researchers are concentrating their time today. There are two key areas being studied today:

  1. The first being understanding where hard-enough problems actually come from. To get an LLM to actually learn, you have to adjust problem difficulty to the so-called “learnable frontier”, which is the Goldilocks zone where the questions aren’t so easy that the model breezes through, but also not so hard the model completely whiffs. There are dozens of studies solely focused on finding the most efficient way to spawn in thousands of these questions in the RL process, and also the best way to automate it.
  2. The second area I noticed researchers spending their time in was understanding the reward mechanism of RLVR, and whether we can actually trust it. Meet the famous concept of “reward hacking”. The problem with training a model via reward is that the model can get smart enough to realize that it can frame its answers to be technically wrong but look correct. One of the most famous examples of this comes from an OpenAI study, where they had a simulated robot hand they wanted to grab a ball and a human would verify it did so. The AI controlling the robot hand learned that it could put its hand in between the ball and the camera from which it was being filmed to give the illusion it was grabbing the ball. Technically wrong but apparently right. Because of this dynamic, I designed my study in a way that detected reward hacking if it were to happen.

Upon reading this literature, a thought slowly began to trickle through my head. I realized that researchers seemed to be tunnel visioned very deeply in these two areas, and that they were being studied in isolation. I then thought to myself, what would happen if we compared these two inputs? If questions and rewards are the two things driving RL outcomes, then why aren’t we studying which one has more leverage in optimizing the RL process? This question catalyzed me to design an experiment and figure it out myself. Over the past month, I went about designing and running a study answering the following question: “Does the reward system or the question system have more leverage on model correctness?” I set out on a quest. I set out to find the bottleneck to the bottleneck.

Designing the experiment aimed at my mystery

In order to test whether the question side or the grader side of RLVR mattered more, I knew I had to make the study as scientifically robust as possible so I could truly isolate the two variables I wanted to measure. I started by picking a basic open-source model that I could run this experiment with. I started with Alibaba’s Qwen 0.5B given it is simple and low-maintenance, but after running a test pilot I realized it was simply not intelligent enough for the purposes of my experiment, so I kicked it up a notch to 1.5B. Then when thinking about how I wanted to test questions versus grader, the two inputs I chose to adjust in the study were question difficulty and grader strictness, which are the primary ways both of these inputs get adjusted in the first place. Since I wanted to isolate two specific variables, I decided to design the study in a 2x2 grid. I would take 4 different copies of the same 1.5B model, and train them 4 different ways. Think of it like a teacher training 4 students for a final exam in 4 different ways: 1): easy questions with loose grading 2) easy questions with strict grading 3) easy+hard questions with loose grading 4) easy+hard questions with strict grading. After training 4 copies of the model in 4 different ways, I would then give each copy the same exact sealed final exam to see which training method resulted in the best score. This would help glean some insight into the training lever that has the highest impact on model correctness, and therefore help us understand what input might be worth investing more into in the RL world of today. For repeatability and mitigation of potential flukes, I ran 3 seeds for each of the 4 scenarios and took the mean for results.

Study design. One model is trained four ways and takes one sealed exam.
task dialeasy-onlyeasy+hardreward dialloosestrictstudent 1×3 seedsstudent 2×3 seedsstudent 3×3 seedsstudent 4×3 seedsall 4 studentsone sealed OOD exam
4 students, 4 homeworks, same exam. The same base model is copied 4 times and trained 4 ways. The reward dial is loose or strict and the task dial is easy-only or easy+hard, giving the 2×2 grid with 3 seeds per cell. Every student then takes the same sealed out-of-distribution final exam.

Architecting the final exam was the most time consuming part of the study, as it would produce the headline results I was actually seeking. Before I ran each of the 4 training methods, I needed to measure how the untrained 1.5B model would perform on the final exam so I would have a baseline score to compare to. After giving the untrained model the exam, I realized that my test was too easy. So I spent a couple of days further tuning the question difficulty so there was a reasonable ceiling for the model to improve against. Think about it this way: if you wanted to measure how the 4 different sets of homework improved exam scores, but all 4 turned up to be 100%, then the ceiling of correctness is already hit and there is no apples-to-apples comparison. Because of this, I increased the difficulty to make the baseline score as low as I possibly could (around mid-to-low teens) and this removed a ceiling limiting visibility for the variables I wanted to isolate.

After tuning the difficulty, I built the scaffolding and architecture of the project, which included both graders and the exam data. I split the training data from the actual test data that I would feed these 4 students during the final exam. In designing the study, I realized that to adhere as best as possible to scientific principles, I needed to measure these 4 students beyond just strict correctness. So I decided on the two other KPIs that I would track in these 4 students' results: format-validity and Pass@k. Format-validity tests whether the agent enters the answer in correct format, and pass@k observes whether the model was correct at all in the allotted attempts for a specific problem. Pass@k is colossal in the RLVR world. In certain cases, researchers have frequently observed models displaying increasing pass@1s (getting it right in one try) and decreasing pass@ks giving the illusion of correctness rising even though answer range is getting worse. This nightmarish scenario was under strict supervision during my study, as this is to some scientists the antichrist of AI training. Even though correct strictness is the headline metric, combining all 3 of these metrics is the best lie detector test I could’ve asked for in a study like this.

For the sealed exam, I standardized it so that each of the 4 students were being measured fairly. I gave each 256 tasks during their training, and when tuning the difficulty I made half of them easy and the other half hard. When developing the answer key, I made sure every run was scored by an always-strict key on a sealed, different-template OOD (out-of-distribution) test set the model never trained on. If I gave it questions that it trained on, that would be like making the final exam exactly like the homework questions. In order to get multiple sets of data to avoid flukes, I ran this experiment with a Task A set and a Task B set. Task A was deal-math, and involved turning messy prose into one computed #, more so testing reasoning capability. Task B was deal-extraction, and involved turning messy prose into a 5-field JSON (company, round, raise, valuation, founders) and more so involved translation. I deliberately ran this with two different skillsets. That way, when tuning each of my knobs I could identify the kinds of questions that each knob is more sensitive to. Finally, within the Task A and Task B set, I made both an easy and a hard band of questions, with the hard band being the headline number I wanted to measure given that the easy band would be very saturated and close to its strict correctness ceiling.

There are two critical concepts underlying my entire training process that made sure everything functioned properly. Without them, my study results would have been nonsense.

  1. Group Relative Policy Optimization (GRPO). I actually needed the Qwen model to learn. There are many different complex learning methods that you can use when running studies, but I chose GRPO because it is the easiest and the cheapest way to train a singular model. GRPO, is a learning method in which when the model has multiple different attempts at one question, GRPO naturally nudges the model towards the swings that are above the average of that group of attempts. So whenever the model takes a swing at a question and it is above the mean for that group it gets told to do more of those attempts, and whenever the model takes swings that are lower than the mean of the group then it gets told to do less of those attempts. This can explain a lot of the variance that occurred when measuring strict correctness in my study.
  2. Low-Rank Adaptation (LoRA). The other critical concept driving my study was LoRA. This was how I fine-tuned my Qwen model using a fraction of the compute. Usually the fine-tuning process takes a ton of raw processing power. But with LoRA, you can fine-tune massive AI models using a fraction of the traditional time and computing power. It freezes the original base model's weights and adds tiny, lightweight "adapter" layers. Only these small adapters are trained, enabling you to teach an AI new skills without retraining the entire model. This is the only reason I was able to run my study front to back on a single Kaggle GPU.

And…

After weeks of testing and haggling the GPU usage limits on Kaggle, the results finally came in:

Task A is deal-math. The chart shows sealed-exam scores by training condition.
strict exact-match %easy bandhard band025507510056.09.0baseline(untrained)79.710.0stricteasy-only69.711.0stricteasy+hard61.09.7looseeasy-only61.012.7looseeasy+hard
Bars show the mean of 3 seeds. Whiskers show ±1 sample standard deviation with ddof=1. The baseline is a single untrained run drawn as an outlined bar with no whisker. The sealed OOD exam has 100 easy and 100 hard items per task. The hard band stays flat in every condition. Task A's hard questions sit above the model's learnable frontier.

Let’s focus on one task at a time, starting with Task A. Unsurprisingly, the baseline strict correctness for Task A naturally fell lower than Task B’s baseline given the task was reasoning heavy. Looking at the grader dial first, it is clear that strict grading materially improved correctness on the easy section of the test. In the untrained run the model scored 92% with loose grading (separate from the strict OOD grader). Because loose grading marks nearly every attempt in a group correct, there is almost no reward variance within the group and GRPO only produces a gradient when attempts in a group earn different rewards. Near-uniform reward means near-zero advantage, so the signal is weak and correctness barely moves. Strict grading splits the group: with a 44% fail rate on the baseline (100% minus 56%), attempts land on both sides of the grader, generating the within-group reward variance GRPO needs. That variance is what moved correctness from 61.0 to 79.7 and 61.0 to 69.7 across the two difficulty knobs. However because this is the near saturated easy section and it already scored 92% on its baseline easy run, there is not much weight I can put on this outcome and analyzing it in depth would be a waste of time. With hard band performance, neither the task difficulty nor the grader knob moved the needle meaningfully, resulting in a null outcome. The hard band questions in Task A were simply just too hard for the model to learn from when graded strictly. Herein lies the lesson of the “learnable frontier”, which is the difficulty that is just out of reach yet attainable for model training, of which Task A was clearly above.

Task B is deal-extraction. The chart shows sealed-exam scores by training condition.
all-5-exact %easy bandhard band025507510078.016.0baseline(untrained)93.719.0stricteasy-only95.059.3stricteasy+hard94.717.3looseeasy-only93.758.3looseeasy+hard
Bars show the mean of 3 seeds. Whiskers show ±1 sample standard deviation with ddof=1. The baseline is a single untrained run drawn as an outlined bar with no whisker. The sealed OOD exam has 100 easy and 100 hard items per task. The metric is all-5-exact %. The hard band jumps only when hard questions enter training, and strict versus loose grading barely changes it.

Looking at Task B, the untrained baseline score is meaningfully higher given how simple extraction is, which helped make the natural ceiling more eye-popping in the easy band. The single most interesting finding lies in the Task B dataset. Hard homework moved the strict grading setting from 19.0 to 59.3, and the loose grading setting from 17.3 to 58.3. There was the answer I was looking for: question difficulty had more leverage when analyzing strict correctness in my study. This is likely because question difficulty is the only lever that actually lets the model practice a hard skill, whereas stricter grading doesn’t infuse new knowledge in the agent. Therefore, a preliminary conclusion is that adjusting task difficulty may carry more leverage in the post-training process when problems are set at the learnable frontier. The reward dial also proved to be null in Task B, consistent with my findings in Task A.

Task B per-field scores on the hard band. Each field shows the baseline and the four trained conditions.
per-field strict %baselinestrict easy-onlyloose easy-onlystrict easy+hardloose easy+hard0255075100companyfoundersvaluationroundraise
This is a per-field diagnostic breakdown, not the headline. The Task B headline is the all-5-exact rate. These are greedy single-decode scores, not pass@k. Bars are the mean of 3 seeds. Whiskers show ±1 sample standard deviation with ddof=1. The baseline is a single untrained run shown as a dashed bar with no whisker.

As I previously touched on, I measured two other KPIs as a sort of bulletproof vest to make sure the nightmarish pass@1 up pass@k scenario wasn’t behind the curtains. Format-validity came in across the board at 100%, meaning the model answered the problem in correct format every single time. This rules out reward hacking in the form of optimizing for format while having the wrong answer, although I guess that would be quite peculiar in such a study as this. The other, more critical KPI that I measured was pass@k, which measures this: in k tries on the 100 questions in this exam, how many questions would it get right if there were multiple attempts per question? Headline strict correctness assumes only pass@1, meaning that the model only gets one attempt per question. The reason pass@1 up and pass@k down is so hellish is because it means the model gets really good at nailing one lucky answer, but when digging past the surface it means its cross-functional intelligence is decreasing and answer range narrowing given because it is gaming towards that one specific answer. The result?

pass@k on the 100 hard-band items. k=4, so each bar is a best-of-4 score. Task B uses its all-5-exact variant.
pass@4 %Task ATask B025507510015.028.0baseline(untrained)18.728.0stricteasy-only17.073.3stricteasy+hard16.327.7looseeasy-only17.372.3looseeasy+hard
Bars show the mean of 3 seeds. Whiskers show ±1 sample standard deviation with ddof=1. The baseline is a single untrained run drawn as an outlined bar with no whisker. The sealed OOD exam has 100 easy and 100 hard items per task. Task A stays in the mid-to-high teens everywhere. Task B moves only when hard questions enter training. pass@1 and pass@4 rise together where the skill is reachable, so there is no diversity collapse.

Luckily, pass@1 and pass@k had a positive correlation the whole study. For Task B hard band easy-only questions, pass@k was stuck at 28, but on Task B hard band easy+hard questions, pass@k jumped to 73. The skill genuinely entered the model’s range rather than its top answer. Having multiple KPIs to point to also helped give me confidence that my finding is a reliable outcome. For the Task A hard band in every condition, pass@k climbed from 15 to just a couple points at 17-19. Again, the ability just wasn't there because Task A questions were not set at the learnable frontier.

The low answer spread in Task A is the ceiling, not evidence of GRPO working. Across multiple attempts the model kept landing in the low teens, so every attempt in the group earned nearly the same reward. With no variance in reward across attempts, there is no advantage and no gradient, which is exactly why progress stalled: GRPO had nothing to learn from. Task B is the contrast case. There the attempts within a group earned different rewards, that spread produced real signals, and the average was able to increase meaningfully when question difficulty was adjusted.

A (hopefully) meaningful data point to progress AI

Why should anyone care about RLVR? The main answer is that in a world where pre-training has hit diminishing returns, the need for increasing model capability will primarily stem from RL, which involves teaching the model genuinely new concepts rather than teaching it to memorize pre-existing information. Given we are leaning on AI to progress the next leg of frontier research and transform fields like medicine, agriculture, and robotics, we need to make sure these models operate at peak functionality, which is increasingly made possible through RL.

Again, there is some genuine debate on whether RLVR actually does teach the model new skills, or whether it just resurfaces existing knowledge. Given that this is a multi-billion dollar industry, this debate is meaningful because it decides where the next billion of capital goes. No one is debating that RL works, but when debating whether there is a higher-leverage post-training method, some people have varying opinions. There is a camp saying RL just resamples, and spits out what it has already digested in pre-training. The second camp believes that if you keep feeding it the right problems it keeps gaining new capability, which justifies continued 10-figure spending in this industry. My study does not answer this—I am simply operating under the assumption that RL is efficient, but it is important to keep in mind from a scientist's perspective nonetheless.

Though my study used a very small model and ran at a rather small scale, I still hope it to be a meaningful data point when thinking about how we can create more leverage in the RL space. And given the J-curve RL is moving along defined by capital pouring in from the skies, it just may be true that the question difficulty half of the equation may warrant more resource allocation than the grader half. Nevertheless, if you haven’t researched RL, I recommend you to. The industry is exploding, and will continue to explode, as it progresses AI research more than any other category meaningfully is at this point in time.

Why did I do this study in the first place? Good question. Rather articulating myself I’ll let the most talented AI researcher of our generation take that burden:

"I am first and foremost a scientist. My goal is to understand nature, but doing science is sort of like reading the mind of God. We humans have these faculties. The world is understandable, but why should it be that way? I think there's a reason computers are just bits of sand and copper. Why should these combine to do anything? I mean, it's absurd. The electrons move around and then that creates an AI system that can defeat a Go master. Why should that be possible? This is beyond evolutionary coincidence. We can build electron microscopes and interrogate reality down to the most minute level. We can build systems that detect black holes colliding from more than a billion years ago. I mean, what is this? What the hell is going on here? I sit at my desk at 2:00 AM and I feel like reality is staring at me, screaming at me, literally screaming at me, trying to tell me something. If I could just listen hard enough, that's how I feel every day. So you can see why I'm trying to build AI. I felt that since I was very young that there's a deep, deep mystery about what's going on here. You can frame it however you want. You can call this God's design, or you can say it's just nature. I'm open minded about the description and I don't know what the answers will turn out to be. At the moment we don't really know what time is or gravity is or any of these things. So there's a mystery waiting to be solved, and it encompasses just about everything I would like to understand. And then I'm perfectly fine to shuffle off my mortal coil."

— Demis Hassabis (Co-Founder of DeepMind)