# Why Another GPU Did Not Make the Training Run Faster > An asynchronous reinforcement-learning case shows how a client-side concurrency limit can hide spare inference capacity. The useful lesson is to follow the queue as the bottleneck moves, while comparing cost and learning outcomes separately. **Published:** September 10, 2026 **Author:** PSEEDR Editorial **Category:** stack **Content tier:** free **Accessible for free:** true **Editorial format:** analysis **News quality eligible:** false **Source count:** 3 **Word count:** 1196 **Read time:** 6 min **Tags:** Reinforcement Learning, Async GRPO, LoRA, vLLM, GPU Capacity Planning **Canonical URL:** https://pseedr.com/stack/why-another-gpu-did-not-make-the-training-run-faster --- Adding another generation GPU is a plausible response when a trainer waits for samples. It helps only if enough work reaches the new GPU. In a [Hugging Face blog post dated September 10, 2026](https://huggingface.co/blog/asyncgrpo-lora-hfjobs), which PSEEDR read on November 1, 2026, Amine Dirhoussi and colleagues describe a run where a third generation replica barely increased throughput. The reason was a setting on the trainer's side, not on the inference servers: it still allowed only 128 generation requests in flight across all replicas. Dividing that allowance across more replicas supplied little additional work. Raising it to 384 exposed capacity the extra machine could use. The case gives infrastructure engineers a more useful question than “How many GPUs should this job have?” At each stage, which component is waiting, and what is preventing its upstream or downstream partner from moving? The answer can change after a successful optimization. A configuration that balanced a slow trainer may constrain a faster one. ## A queue connects two different jobs Group Relative Policy Optimization, or GRPO, trains on groups of model-generated answers and their rewards. In an asynchronous implementation, producing those answers and updating the model overlap. The [current TRL trainer documentation](https://huggingface.co/docs/trl/main/en/async_grpo_trainer) describes a background rollout worker that generates and scores completions, then puts training samples into a queue. The training loop removes samples and updates the weights. That documentation has evolved since the September run, so it supports the mechanism described here rather than fixing which features existed at the time. Because the two sides meet at that queue, how long each side waits tells an engineer which side is slow. If the queue stays full while the trainer rarely waits for samples, generation is already ahead, and the rollout worker must wait for room before it can add more. That waiting is backpressure. If the queue empties and the trainer spends time waiting for rollouts, sample production is insufficient for the current consumption rate. TRL exposes timings for these states, including rollout wait and backpressure. It also limits sample age with `max_staleness`, because a queued completion can have been generated by an older model version. If a larger queue lets samples wait longer while the policy keeps changing, more of them can fall outside the age limit and be discarded by the trainer. This is why enlarging the queue is not simply a free way to absorb every imbalance. The observations need to be read together. A full queue at one instant could be a temporary burst. Sustained fullness, low trainer wait and sustained backpressure describe a different condition. The combination points toward the trainer, which consumes the samples, rather than toward generation. Likewise, an empty queue is not itself proof that generation needs another GPU: a concurrency limit, reward calculation or another upstream delay may be withholding work from capacity that already exists. This reasoning is a diagnosis method, not a universal threshold. The relevant pattern depends on the run's timing, sample sizes and synchronization schedule. An engineer should be able to explain what a proposed change is expected to do to the queue before paying for a larger run. ## Smaller updates make separation practical The [companion implementation](https://github.com/AmineDiro/hfjobs-lora-buckets) separates a trainer job from multiple vLLM serving jobs. It transfers LoRA adapters through a shared storage-bucket mount. LoRA keeps the base model fixed while learning a smaller set of parameters, so the serving replicas can load updated adapters instead of receiving a full model on each update. A proxy makes those replicas appear as one endpoint to the trainer. It routes generation requests, adds the required authentication, and broadcasts adapter loads and state-changing requests to the replicas. The repository's trainer publishes each new adapter under a versioned path rather than overwriting one file, so a policy name always refers to one set of weights. Separating the jobs changes what must be coordinated. A request must reach a worker with the intended policy version. A load operation reaching one replica is insufficient if later requests can land on another. The apparent simplicity of one serving URL therefore depends on the proxy preserving a common meaning for the requested version across its workers. This is distinct from splitting an ordinary serving system into prefill and decode pools. Here, generation produces training data, while training changes the policy that will generate subsequent data. Faster movement around the loop can change the age and quantity of those samples as well as the elapsed time. ## Successful optimization changes the next decision Suppose a training loop becomes faster while generation stays unchanged. This is an illustrative pipeline, not a new benchmark. Initially, the queue may remain full and the trainer may determine the completion rate. As the trainer consumes samples faster, the queue begins to drain. Eventually the draining queue can expose a generation limit that was present all along but previously hidden by backpressure. At that point, the earlier diagnosis is stale. Adding generation capacity can now be reasonable, even though it would have been wasted before. But the client must be able to submit enough concurrent work to use it. A limit on requests in flight is shared across all replicas; adding workers does not automatically increase the amount of work admitted into the system. The converse is equally important. Increasing concurrency until the queue fills again does not mean the larger value is optimal. Samples now spend longer waiting, and more of the work may come from older policies. The objective is a useful rate of learning within the accepted freshness and resource budget, not the fullest possible queue. For this reason, performance reports should preserve both the causal sequence and the resource accounting. “We changed batching, then generation became limiting” is more actionable than a single multiplier. It lets another team test whether its own symptoms match before copying the configuration. ## A faster run is not yet an equal-cost result The authors' [final comparison](https://huggingface.co/blog/asyncgrpo-lora-hfjobs) reports 500 steps falling from 3 hours 27 minutes to 53 minutes. But generation replicas increased from two to three, and trained samples rose from 64,000 to 84,078. The final 20-step mean rewards were 0.438 and 0.416. Those differences prevent treating the headline as an equal-hardware, equal-sample speedup with identical final quality. That does not erase the engineering improvement. It tells the reader what the comparison establishes: this revised system completed its reported run faster under its revised configuration. A purchasing decision needs the corresponding total resource use; a training decision needs a consistent quality target and evaluation protocol. A team can separate these questions without turning every tuning run into a large research project. Short diagnostic runs can test whether queue behavior changes as expected. A selected configuration then needs a longer comparison that records what was actually trained, the resources used and the achieved outcome. Equal optimizer-step counts alone do not make two runs equivalent when samples per step differ. The third GPU was not inherently useless. Its usefulness depended on the work admitted to it and on the speed of the stage consuming its output. Following those dependencies gives engineers a way to decide what to change next. A lower wall-clock time is the result to explain, not a substitute for that explanation. --- ## Sources - https://huggingface.co/blog/asyncgrpo-lora-hfjobs - https://github.com/AmineDiro/hfjobs-lora-buckets - https://huggingface.co/docs/trl/main/en/async_grpo_trainer