A CNN developed by a 10-year-old researcher outperformed GPT-4o and Claude 3.5 in a Lynch Lab test of skin lesion image classification. The result offers engineers an important reminder: for medical tasks, data, specialized architecture, and evaluation design may matter more than model size.
Original study: Lynch Lab — LLM Skin Cancer.
What the reported results show
Lynch Lab compared three types of models: a CNN developed by the researcher, GPT-4o, and Claude 3.5. Each model was evaluated on skin lesion image classification.
The study page reports that the CNN performed better than the frontier LLMs included in the test.
Two conclusions need to be separated.
First, the report says that the CNN produced better classification results for this particular task and evaluation setup.
Second, that does not establish that CNNs are generally superior to LLMs, much less that small models have replaced large ones. The original page does not provide the full test-set size, sensitivity, specificity, or evaluation protocol, nor does it include enough clinical data to support broad conclusions.
Accordingly, the claim that “the CNN beat frontier LLMs” applies only to the task and setup evaluated in this report.
What the comparison actually tested
The work did not simply ask ChatGPT to describe a lesion photo and then judge, from its written response, whether it looked like skin cancer.
Instead, it evaluated the CNN and frontier LLMs on skin lesion classification. The models processed similar images and produced outputs that could be reduced to comparable classifications.
This comparison has at least three advantages:
- It uses a unified task rather than comparing CNN classification metrics with LLM text responses.
- It directly compares models, making it possible to examine differences in architecture and training objectives.
- It produces measurable results rather than relying on a handful of demonstrations.
It still does not amount to complete clinical validation. Medical AI cannot be assessed through accuracy alone.
Missing a malignant lesion and incorrectly flagging a benign one carry different costs. Before a system enters research or deployment, its sensitivity, specificity, confidence calibration, performance on external datasets, and behavior within clinicians' actual workflows must also be examined.
The finding is useful for raising questions, but not for making clinical decisions.
Why a CNN may fit this task better
GPT-4o and Claude 3.5 are general-purpose multimodal models, but broad capabilities do not automatically transfer to every specialized task.
A dedicated CNN can be trained around skin lesion recognition, including its training data, label space, and loss function. It learns visual features and the classification target directly without first generating a passage of natural language.
A general LLM has broader objectives: understand images, follow instructions, and generate text. The more fixed the task, the more likely that generality becomes an additional burden.
| Dimension | Dedicated CNN | General multimodal LLM |
|---|---|---|
| Task boundary | Fixed label set and explicit objective | Handles multiple tasks through instructions |
| Training signal | Optimized directly for the target labels | More diffuse objectives, including visual understanding and language generation |
| Output space | Classes or probability distributions | Natural language and open-ended answers |
| Inference cost | Typically lighter and suitable for local deployment | More dependent on computing resources and API services |
| Interpretability | Can be analyzed with saliency maps and intermediate layers | A textual rationale is not causal evidence |
The final row requires particular caution. When an LLM says that a lesion “may be melanoma,” that does not mean it has supplied reliable medical grounds for the judgment.
Medical settings also require consideration of latency, service cost, and availability. API-based models depend on network access and provider services. A dedicated CNN may be smaller and faster while also running locally at a clinic. For high-volume preliminary screening, system cost can sometimes matter more than a small lead on a leaderboard.
A larger model is therefore not automatically better for every medical task.
The real threshold for a 10-year-old developer
A low barrier to entry is not the same as no barrier.
For a CNN to participate in the comparison, someone still had to organize and annotate the data, train the model, divide the training and test sets, call the frontier LLMs, parse their results, and standardize the evaluation criteria.
Study entry point: Lynch Lab — LLM Skin Cancer.
The work received 228 points and 112 comments on Hacker News. Discussion page: HN Comments.
These figures come from the original material. Commenters focused on the research process, data sources, evaluation design, and the limits of the conclusions. The discussion's value lies not simply in the contrast between the researcher's age and those of more established developers, but in subjecting the project to closer methodological scrutiny.
The most important information still needed includes:
- The number of images in the test set, the institutions that supplied them, and whether any duplicates or data leakage existed;
- The distribution of disease types, skin tones, and image quality;
- Whether the CNN's training data overlapped with the data used to test the LLMs;
- How sensitivity, specificity, precision, and F1 were calculated;
- Whether the LLMs received image input and whether repeated runs produced consistent results;
- Whether the results were reproduced on an independent external dataset.
Publishing these details would be more useful than adding an even more eye-catching headline.
How medical AI comparisons should be conducted
If I were to reproduce or extend the evaluation, I would proceed in the following order.
1. Fix the task and label space
First, establish whether the inputs are isolated lesion photographs, clinical photographs, or dermoscopy images.
A model may output natural language, but those outputs must ultimately be mapped to the same set of classes. Otherwise, the comparison measures expression style rather than classification ability.
2. Standardize data and prompting conditions
Every model should use the same test set.
For the LLMs, the evaluation should also record the model version, temperature, system prompt, number of samples, and whether images were compressed or cropped. Both GPT-4o and Claude 3.5 may have changed through product updates. Without those conditions, the results will be difficult to reproduce.
3. Report more than accuracy
A medical classification evaluation should also report:
- Sensitivity;
- Specificity;
- Precision, recall, and F1;
- Calibration error and prediction confidence;
- Grouped performance across disease types, skin tones, and imaging devices.
If the task involves preliminary screening, thresholds should reflect the costs of missed diagnoses and false positives. Metrics cannot be interpreted independently of their intended use.
4. Calculate system costs separately
Record latency per inference, GPU or API costs, throughput, model size, and deployment dependencies.
Dedicated CNNs and closed LLMs have different cost structures. Comparing only the price of one API call would omit local inference, maintenance, and fallback costs.
5. Inspect explanations and failure cases
Manually review high-confidence correct predictions, high-confidence errors, and low-confidence samples.
Pay particular attention to whether the models rely on spurious cues such as colostomy borders, imaging equipment, annotations, or background color. A specialized model that outputs a fixed class may appear more concise, but brevity is not itself evidence.
Editorial perspective
The significance of this result lies not in a “child overtakes the adults” narrative, but in its return of model selection to the task itself. Medical image classification typically has fixed inputs, fixed labels, and quantifiable costs for different errors. A dedicated CNN can train directly around those constraints, while a general LLM covers a wider range of tasks and may require more expressive output and greater inference overhead.
My expectation is that medical AI will increasingly combine general-purpose and specialized models. A general model can organize patient histories, draft reports, and support interaction, while a specialized vision model handles stable, measurable classification. Using each model for the tasks it handles best is more practical than asking one model to perform every step.
This differs from earlier arguments over demonstration-style benchmarks. At least here, the CNN and frontier LLMs were placed within the same classification task instead of being judged on which produced more fluent prose. However, the original material lacks the test-set size, sensitivity, specificity, full protocol, and external clinical validation. It is a result worth verifying, not yet a clinical conclusion. The next question is whether independent data reproduces it, rather than whether the “10-year-old developer” label alone is remarkable.