A Cheap Fine-Tune Still Needs an Honest Baseline and a Stopping Point

A citrus-image model improved from 14.9 percent to 52.8 percent on a test of 335 examples, after a training run whose reported compute bill was about $1.90. That is a striking result for a small experiment. It becomes more useful once the developer knows what the test counted.

The model card defines the primary score as a text check: whether the class name that the dataset's knowledge base assigns to the image, such as "Citrus canker", appears in the generated answer. A correct explanation that does not repeat that name counts as a miss under this matcher. The card also reports a more permissive score, but the list of alternative names it accepts was adjusted on those same evaluation answers. That makes the relaxed score a secondary, optimistic result rather than a fresh test.

This is one of the experiments described by Yuvraj Sharma and Abubakar Abid in Hugging Face's October 8 account of building models with ML Intern. ML Intern, Hugging Face's training agent, can assemble data, run training jobs and publish artifacts. The developer still has to decide what improvement means and when another job is worth running.

That decision has two parts: establish a comparison whose score has a clear meaning, then preserve enough intermediate work to choose a useful stopping point. Neither follows automatically from a successful training command.

A baseline needs the same question and the same ruler

The citrus project fine-tuned Qwen3.5-2B for questions about citrus plants. Running the original model on the same evaluation examples gave the project a starting point. Without that baseline, a trained model's score could not show whether training improved the chosen task at all.

Yet a common ruler can still measure only part of the intended result. Here the strict test checks for a class name in the answer. The card explains that 42 of the 335 test questions already name the problem and ask only about treatment, so an answer can respond appropriately without repeating the label the matcher expects. It also describes healthy-image answers that use a no-symptoms phrase rather than the required label "Healthy citrus tissue", which is why the healthy class scores zero under the strict matcher. Those omissions lower the strict score without, by themselves, establishing a diagnostic mistake.

The reverse distinction matters too. Including a class name does not establish that the answer's explanation or treatment suggestion is correct. The score supports a statement about that specific output check. It cannot carry every claim a product might make about the answer.

Publishing the strict metric together with its limitations, as the model card does, is therefore useful. A developer can see a comparable before-and-after result while identifying where the test disagrees with the intended task. The next step is to inspect those disagreements, not to replace the original score silently with whichever number looks better.

Repairing the ruler changes the comparison

The model card's relaxed evaluation accepts additional names and phrases, such as pathogen names and distinctive knowledge-base wording. That can recover answers a literal matcher wrongly rejects. But the card says the alias table was tuned on this evaluation set. The relaxed result describes a closer reading of known answers. It does not provide the same evidence as a rule fixed before a new test set is seen.

For a developer running a similar project, this suggests a practical separation. Use one set of examples to discover that a matcher is too rigid, decide what equivalent answers should count, and freeze that rule. Then apply it to examples that did not shape the revision. This is an evaluation recommendation from PSEEDR, not an additional experiment reported in the ML Intern account.

The distinction matters most when the same agent writes both the training pipeline and the evaluation script, as the citrus repository's bundled scripts suggest happened here. If the evaluation checks only that the pipeline produces the expected output format, a run can deliver consistent files and attractive scores while leaving the product question unanswered. The developer's brief to the agent should therefore specify the output being judged, the comparator and the cases the model must keep handling. Asking for a baseline helps only if the baseline and the trained model face that same decision.

The published card lets a reviewer separate the measured gain from a product claim for the citrus example. It does not establish that the model is ready for agricultural use, and the card itself says treatment text is templated from a knowledge base and should be reviewed with an agronomist before real decisions. The point for an ML developer is how to read a training result before turning it into a capability claim.

A smoke test answers a smaller question

For the image LoRAs, small adapters trained on top of an existing image model, Sharma describes asking ML Intern to run 50 training steps and then check that the saved weights had actually changed before paying for the full run. This catches a different problem from a bad evaluation metric: the intended training operation might not have happened.

Changed weights are evidence that parameters were updated. They are not evidence that the update improved the task. A useful small run therefore checks the handoff as well as the optimizer: can the saved artifact be loaded in the intended way, and does the evaluator operate on it rather than on an unchanged base model? Those two load-and-evaluate checks are PSEEDR's suggested additions, not steps the blog reports. The blog also does not establish a universal test length that works for every training method.

The small run puts failures in a short job that precedes the full one, where they are easier to attribute. The blog reports projects with many jobs, including resubmissions after missing packages or incorrect paths; a camera-angle LoRA, another project in the same account, took 48 jobs counting those failures. That matters when interpreting a low final bill. Automation can carry out the retries, but the cost and elapsed time of reaching a usable artifact remain part of the experiment.

Save the checkpoint that does the job

The clearest stopping example comes from a separate LoRA trained to draw Huggy, the Hugging Face mascot. In the blog's account, checkpoints were saved every hundred steps and compared using the same prompts. The intended character style was already present at step 200. From step 500, the author reports, the style began appearing in prompts unrelated to the character.

That is the author's qualitative comparison, not a numerical benchmark reproduced by PSEEDR. The linked model repository did not provide a model card when checked on November 1. The account still illustrates a concrete reason to retain intermediate checkpoints: further training can strengthen the desired specialization while spreading it into requests where it is unwanted.

A stopping rule must therefore include both the requested behavior and behavior that should stay unchanged. For an image adapter, that can mean character prompts alongside unrelated scenes. For an extraction model, it can mean valid fields alongside cases where no field should be returned. Those are proposed comparison sets, not results from either project.

The blog also describes a second selection by comparison. For the doodle-in LoRA, which replaces a scribble on a photo with a named object, the saved checkpoints were compared on 48 test pairs and step 500 was chosen. Selecting against a fixed set creates its own risk: the chosen checkpoint can fit that set better than future inputs. The blog does not say how those 48 pairs relate to the final test set, so PSEEDR draws no conclusion about that project. The general point stands. A separate final test remains useful after the training settings and checkpoint are chosen. More saved artifacts make a choice possible; they do not make every choice unbiased.

Buy evidence with the next training run

The examples come from a platform promoting its own training agent, and they are selected projects rather than a measured success rate for all ML Intern attempts. The blog's cost table covers the CPU and GPU job charges reported for each session. It is not a complete price for staff time, every external service or operating the resulting model.

Within that boundary, the lesson is concrete. A baseline shows what changed under a stated measure. A short run checks that the intended process produces a usable artifact. Intermediate checkpoints let the developer stop when the requested gain arrives, before an unwanted change becomes acceptable merely because training finished.

Before paying for another run, name the uncertainty it should resolve. If the score's meaning is unclear, another epoch will not repair the evaluation. If the task already works and unrelated behavior is getting worse, the earlier checkpoint may be the better result. A small compute bill makes experimentation accessible. A comparison tied to the product decision makes it informative.