ClinicalBench logo

ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?

TLDR: We discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making.

Accepted at ACM SIGKDD (KDD) 2026

1. Northwestern University 2. University of Texas at Austin 3. Mass General Brigham & Boston Children's Hospital, Harvard Medical School 4. Imperial College London 5. Ohio State University 6. Massachusetts General Hospital, Harvard Medical School 7. University of Minnesota 8. Weill Cornell Medicine, Cornell University 9. Emory University
* Equal contribution.  † Corresponding author.
ClinicalBench framework: three clinical prediction tasks over MIMIC-III and MIMIC-IV, dataset filtering and division, code-to-text conversion, benchmarked traditional ML models and LLMs, and evaluation with Macro F1, AUROC, confusion matrices and loss curves
Overview of ClinicalBench.

Abstract

Large Language Models (LLMs) hold great promise to revolutionize current clinical systems for their superior capacities on medical text processing tasks and medical licensing exams. Meanwhile, traditional ML models such as SVM and XGBoost have still been mainly adopted in clinical prediction tasks. An emerging question is: Can LLMs beat traditional ML models in clinical prediction? Thus, we build a new benchmark ClinicalBench to comprehensively study the clinical predictive modeling capacities of both general-purpose and medical LLMs, and compare them with traditional ML models. ClinicalBench embraces three common clinical prediction tasks, two databases, 14 general-purpose LLMs, 8 medical LLMs, and 11 traditional ML models. Through extensive empirical investigation, we discover that both general-purpose and medical LLMs, even with different model scales, diverse prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet, shedding light on their potential deficiency in clinical reasoning and decision-making. We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be utilized to bridge the gap between LLMs' development for healthcare and real-world clinical practice.


The Benchmark

ClinicalBench at a Glance

3
clinical prediction tasks
2
clinical databases
14
general-purpose LLMs
8
medical LLMs
11
traditional ML models
4
prompting strategies
2
fine-tuning strategies

Tasks

  • Length-of-Stay Prediction. Three-class classification: < 1 week, 1–2 weeks, or > 2 weeks. Scored with Macro F1.
  • Mortality Prediction. Binary. Will the patient decease during the current visit? Scored with F1.
  • Readmission Prediction. Binary. Is the patient at high risk of readmission within a given time frame? Scored with F1.

Following PyHealth, every task predicts from demographics (gender, age) plus diagnoses, procedures, medications and drugs for the current visit. AUROC is also reported.

Databases & Processing

  • MIMIC-III v1.4 (ICU admissions, 2001–2012) and MIMIC-IV v2.2 (2008–2019), both from Beth Israel Deaconess Medical Center.
  • Filtering: first-time visits only, patients aged 18+, no missing values.
  • Division: stratified split into train / validation / test; under-sampling rebalances the training set only, leaving validation and test at their real-world label distribution.
  • Code-to-text conversion: ICD-9/ICD-10 diagnosis and procedure codes and ATC codes are rendered into natural language so LLMs can use their language understanding.

Benchmarked Methods

11 traditional ML models

XGBoostLogistic RegressionDecision TreeRandom ForestAdaBoostSVMNaive BayesKNNMLPTransformerRNN

14 general-purpose LLMs

Llama3-8BLlama3-70BMistral-v0.3-7BGemma2-9BQwen2-0.5BQwen2-1.5BQwen2-7BYi-v1.5-6BYi-v1.5-9BYi-v1.5-34BVicuna-v1.5-7BPhi3.5-mini-3.8BInternLM2.5-7BMiniCPM3-4B

8 medical LLMs

Meditron-7BMeditron-70BMedllama3-8BBioMistral-7BMed42-8BMed42-70BBioMedGPT-7BInternist-7B

Only open-source LLMs are evaluated: closed-source models such as GPT-4 are prohibited on MIMIC-III and MIMIC-IV under the PhysioNet credentialed data use policy.


Results

Research Questions and Findings

Click a finding to expand the evidence behind it.


Finding 1 RQ 1 · Can directly prompting LLMs beat traditional ML models?
Directly prompting general-purpose and medical LLMs, even with different decoding temperatures or parameter scales, cannot beat traditional ML models in clinical prediction.

We compare 9 general-purpose and 6 medical LLMs at around the 7B scale against 11 traditional ML models, then push further by sweeping decoding temperatures and scaling parameters up to 70B.

Length-of-Stay · Macro F1
SVM70.28
Best LLM33.43
Best LLM is Meditron-7B, 36.9 points behind.
Mortality · F1
XGBoost65.75
Best LLM43.03
Best LLM is Gemma2-9B, 22.7 points behind.
Readmission · F1
SVM20.17
Best LLM13.72
Best LLM is BioMedGPT-7B, essentially the 13.64 floor of always predicting the minority class.

Directly prompting on MIMIC-III, averaged over five runs.

EvidenceHead-to-head results on both databases
Method Length-of-Stay Prediction Mortality Prediction Readmission Prediction
Macro F1 (%)AUROC (%) F1 (%)AUROC (%) F1 (%)AUROC (%)
Majority 23.37(23.37, 23.37) 50.00(50.00, 50.00) 0.00(0.00, 0.00) 50.00(50.00, 50.00) 0.00(0.00, 0.00) 50.00(50.00, 50.00)
Minority 10.72(10.72, 10.72) 50.00(50.00, 50.00) 20.88(20.88, 20.88) 50.00(50.00, 50.00) 13.64(13.64, 13.64) 50.00(50.00, 50.00)
Traditional ML Models
XGBoost 67.94(67.87, 68.01) 93.83(93.78, 93.88) 65.75(65.56, 65.94) 95.97(95.93, 96.01) 19.92(19.75, 20.09) 69.24(68.75, 69.73)
Logistic Regression 66.52(66.43, 66.61) 93.09(92.99, 93.19) 63.09(62.96, 63.22) 94.59(94.53, 94.65) 19.88(19.73, 20.03) 69.19(68.85, 69.53)
Decision Tree 59.14(59.04, 59.24) 85.22(85.14, 85.30) 52.17(52.05, 52.29) 84.18(83.93, 84.43) 16.1(15.94, 16.26) 57.57(57.05, 58.09)
Random Forest 61.14(61.09, 61.19) 91.23(91.15, 91.31) 54.23(54.10, 54.36) 91.65(91.56, 91.74) 18.78(18.64, 18.92) 69.3(69.04, 69.56)
AdaBoost 66.6(66.48, 66.72) 92.91(92.85, 92.97) 61.19(60.99, 61.39) 93.62(93.54, 93.70) 17.92(17.75, 18.09) 64.51(64.20, 64.82)
SVM 70.28(70.20, 70.36) 94.77(94.71, 94.83) 63.97(63.81, 64.13) 95.69(95.65, 95.73) 20.17(20.03, 20.31) 71.74(71.36, 72.12)
Naive Bayes 42.93(42.84, 43.02) 61.55(61.43, 61.67) 23.43(23.41, 23.45) 56.85(56.80, 56.90) 16.86(16.77, 16.95) 58.76(58.56, 58.96)
KNN 44.27(44.20, 44.34) 73.72(73.59, 73.85) 44.52(44.37, 44.67) 85.39(85.34, 85.44) 15.09(14.91, 15.27) 59.45(58.73, 60.17)
MLP 63.54(63.47, 63.61) 90.61(90.52, 90.70) 59.32(59.16, 59.48) 93.01(92.92, 93.10) 18.86(18.69, 19.03) 67.47(67.22, 67.72)
Transformer 63.56(63.45, 63.67) 89.58(89.50, 89.66) 57.38(57.06, 57.70) 93.35(93.26, 93.44) 18.97(18.81, 19.13) 67.27(67.01, 67.53)
RNN 64.23(64.12, 64.34) 90.08(90.01, 90.15) 59.03(58.74, 59.32) 91.97(91.89, 92.05) 19.40(19.26, 19.54) 69.32(69.12, 69.52)
General-purpose Large Language Models
Llama3-8B 25.78(25.72, 25.84) 83.82(83.74, 83.90) 25.81(25.78, 25.84) 85.40(85.33, 85.47) 13.64(13.64, 13.64) 43.00(42.50, 43.50)
Mistral-v0.3-7B 11.59(11.57, 11.61) 69.73(69.64, 69.82) 38.11(38.06, 38.16) 86.84(86.80, 86.88) 13.44(13.38, 13.50) 47.16(46.69, 47.63)
Gemma2-9B 17.30(17.26, 17.34) 79.19(79.08, 79.30) 43.03(42.96, 43.10) 86.46(86.41, 86.51) 13.66(13.66, 13.66) 47.14(46.87, 47.41)
Qwen2-7B 12.17(12.14, 12.20) 73.97(73.82, 74.12) 34.98(34.81, 35.15) 85.97(85.91, 86.03) 2.14(1.88, 2.40) 44.04(43.61, 44.47)
Yi-v1.5-9B 10.89(10.88, 10.90) 74.51(74.41, 74.61) 28.75(28.72, 28.78) 76.05(75.96, 76.14) 11.35(11.23, 11.47) 45.02(44.76, 45.28)
Vicuna-v1.5-7B 22.05(22.01, 22.09) 73.20(73.11, 73.29) 20.91(20.91, 20.91) 77.90(77.80, 78.00) 13.64(13.64, 13.64) 39.51(39.18, 39.84)
Phi3.5-mini-3.8B 13.17(13.14, 13.20) 73.67(73.63, 73.71) 26.45(26.43, 26.47) 80.37(80.30, 80.44) 12.40(12.30, 12.50) 46.64(46.37, 46.91)
InternLM2.5-7B 11.53(11.52, 11.54) 82.38(82.28, 82.48) 33.89(33.57, 34.21) 83.89(83.81, 83.97) 0.00(0.00, 0.00) 45.84(45.51, 46.17)
MiniCPM3-4B 22.25(22.21, 22.29) 75.95(75.86, 76.04) 24.01(24.00, 24.02) 86.28(86.21, 86.35) 13.36(13.31, 13.41) 42.23(41.95, 42.51)
Medical Large Language Models
Meditron-7B 33.43(33.39, 33.47) 76.63(76.51, 76.75) 20.88(20.88, 20.88) 77.26(77.20, 77.32) 13.64(13.64, 13.64) 44.93(44.43, 45.43)
Medllama3-8B 14.88(14.85, 14.91) 78.27(78.22, 78.32) 24.12(24.11, 24.13) 78.74(78.66, 78.82) 13.64(13.64, 13.64) 49.80(49.24, 50.36)
BioMistral-7B 21.96(21.89, 22.03) 40.94(40.73, 41.15) 30.53(30.45, 30.61) 77.88(77.81, 77.95) 13.37(13.28, 13.46) 40.93(40.54, 41.32)
Med42-8B 12.82(12.80, 12.84) 84.76(84.70, 84.82) 28.02(27.98, 28.06) 82.55(82.46, 82.64) 13.62(13.60, 13.64) 42.95(42.56, 43.34)
BioMedGPT-7B 14.11(14.11, 14.11) 61.54(61.45, 61.63) 22.55(22.54, 22.56) 78.93(78.82, 79.04) 13.72(13.65, 13.79) 42.89(42.67, 43.11)
Internist-7B 11.71(11.68, 11.74) 81.10(81.04, 81.16) 37.97(37.92, 38.02) 87.25(87.20, 87.30) 12.07(11.90, 12.24) 47.67(47.36, 47.98)

Performance of LLMs and traditional ML models on the three clinical prediction tasks. Average over five runs. The numbers in brackets are the 95% confidence interval computed from the five runs. (Macro) F1 and AUROC are used because of label imbalance. Darker and shading indicates scores closer to 100%. Majority / Minority denote always predicting the majority / minority class. Results should be interpreted keeping in mind that ICD-based features are administrative rather than purely clinical data.

EvidenceHow the predictions actually fail
Confusion matrices for XGBoost, Transformer, Llama3-8B and Meditron-7B on Mortality Prediction in MIMIC-III. The two traditional ML models balance both classes, while Llama3-8B and Meditron-7B push almost every patient into the positive class.
Confusion matrices on Mortality Prediction (MIMIC-III). XGBoost and Transformer balance precision and recall, whereas Llama3-8B and Meditron-7B flag nearly every patient as positive. Meditron-7B assigns the positive class to all 5,146 test patients.
EvidenceDecoding temperature
Macro F1 of 15 LLMs across decoding temperatures 0 to 1 on Length-of-Stay Prediction in MIMIC-III, all far below the dashed lines marking SVM, XGBoost, RNN and Transformer
Decoding temperature vs. Macro F1 on Length-of-Stay Prediction (MIMIC-III). Raising the temperature helps some models and hurts others, but no setting brings any LLM near the dashed traditional-ML reference lines. Note the broken y-axis.
EvidenceParameter scaling
Method Length-of-Stay Prediction Mortality Prediction Readmission Prediction
MIMIC-IIIMIMIC-IV MIMIC-IIIMIMIC-IV MIMIC-IIIMIMIC-IV
Traditional ML Models
XGBoost 67.98 64.19 64.46 48.15 19.33 27.73
SVM 70.63 66.43 63.10 46.27 18.85 28.27
Transformer 64.11 54.97 56.46 42.81 17.80 26.40
RNN 64.76 59.89 56.32 48.9 19.96 25.58
General-purpose Large Language Models
Llama3-8B 25.38 17.73 26.06 15.12 13.64 25.05
Llama3-70B 18.66 23.01 30.33 18.43 12.40 22.81
Qwen2-0.5B 5.11 5.78 1.94 3.11 0.00 0.60
Qwen2-1.5B 23.37 29.55 20.91 6.88 13.72 24.82
Qwen2-7B 11.96 6.48 35.65 43.53 4.26 0.29
Yi-v1.5-6B 25.81 30.18 28.35 18.26 13.69 24.66
Yi-v1.5-9B 10.94 6.96 28.82 16.55 10.74 20.73
Yi-v1.5-34B 18.92 18.42 41.78 33.63 13.46 24.12
Medical Large Language Models
Meditron-7B 33.46 26.90 20.88 6.70 13.64 24.92
Meditron-70B 30.15 35.67 43.16 47.93 8.67 9.12
Med42-8B 12.69 9.97 28.59 16.79 13.59 25.06
Med42-70B 15.43 22.95 42.96 31.21 12.99 23.87

Parameter scaling within model series. Length-of-Stay uses Macro F1 (%), the other two tasks use F1 (%). Scaling from 0.5B to 70B does not reliably help. Qwen2-1.5B beats both Qwen2-0.5B and Qwen2-7B on Length-of-Stay and Readmission, and Yi-v1.5-6B beats the 9B and 34B models on Length-of-Stay in both databases.

What we observe
  • Traditional ML models generally outperform both general-purpose and medical LLMs at a moderate scale, across all three tasks and both databases.
  • The advantage is highly statistically significant. The lower bounds of the traditional ML models' 95% CIs mostly sit above the upper bounds of the LLMs'.
  • Traditional ML models keep their advantage in low-resource settings, staying stable and still ahead of LLMs when trained on just 5%, 10%, 20% or 40% of the training set.
  • Adapting general-purpose LLMs to the medical domain may not improve, and can even hurt, clinical reasoning. Med42-8B beats its base Llama3-8B on MedQA and MedMCQA, yet drops clearly on Length-of-Stay Prediction.
  • LLM prediction patterns differ from those of traditional ML models. Under-sampling lets traditional models balance precision and recall, while LLM predictions can be overwhelmingly biased toward one class, producing high false-positive rates.
  • Raising the decoding temperature affects LLMs in opposite directions. Internist-7B improves steadily as the temperature rises, while Meditron-7B degrades.
  • Temperature tuning alone never reaches typical traditional ML models such as XGBoost, SVM, RNN and Transformer.
  • Parameter scaling does not necessarily lead to better clinical prediction performance.
  • Scaling can even hurt. Yi-v1.5-6B performs much better than Yi-v1.5-9B and Yi-v1.5-34B on Length-of-Stay Prediction in both databases.
  • Scaling sometimes does help. Meditron-70B improves substantially over Meditron-7B on Mortality Prediction, but a large gap to traditional ML models remains.
Finding 2 RQ 2 · Can LLMs with prompt engineering beat traditional ML models?
The effectiveness of typical prompting engineering techniques is generally limited and they still cannot make general-purpose and medical LLMs beat traditional ML models.

Zero-shot Chain-of-Thought, Self-Reflection, Role-Playing and In-Context Learning are applied to a general-purpose series (Llama3) and a medical series (Meditron), each at both the 7–8B and 70B scales.

EvidenceFour prompting strategies on two model series
Method Length-of-Stay Prediction Mortality Prediction Readmission Prediction
MIMIC-IIIMIMIC-IV MIMIC-IIIMIMIC-IV MIMIC-IIIMIMIC-IV
Traditional ML Models
XGBoost 63.88 57.25 48.04 40.00 18.80 26.95
SVM 66.16 59.08 56.47 36.59 18.48 26.52
Transformer 61.49 53.39 51.85 29.13 17.32 23.21
RNN 64.26 60.09 58.96 49.80 20.80 26.24
General-purpose Large Language Models
Llama3-8B 26.36 16.14 25.84 14.35 13.78 25.40
+ Chain-of-Thought 14.76 9.22 23.18 10.81 12.05 22.90
+ Self-Reflection 25.88 17.53 24.10 14.29 15.70 22.80
+ Role-Playing 27.44 17.30 25.44 14.11 13.78 24.87
+ In-Context Learning 51.77 54.72 28.02 14.85 11.97 24.46
Llama3-70B 18.88 21.76 29.82 17.80 13.72 22.03
+ Chain-of-Thought 10.00 9.02 0.00 0.00 0.00 0.00
+ Self-Reflection 25.70 26.28 21.13 7.28 13.81 25.13
+ Role-Playing 11.68 11.37 23.34 11.68 13.83 24.95
+ In-Context Learning 41.04 47.52 25.22 12.41 12.47 25.43
Medical Large Language Models
Meditron-7B 35.88 27.53 20.79 6.58 13.78 24.87
+ Chain-of-Thought 16.04 12.58 13.90 5.95 12.83 24.34
+ Self-Reflection 18.36 10.40 20.47 6.60 13.78 24.91
+ Role-Playing 27.73 17.54 20.79 6.58 13.78 24.87
+ In-Context Learning 14.09 14.62 23.72 11.37 11.64 23.27
Meditron-70B 27.23 34.52 46.15 34.48 9.64 9.90
+ Chain-of-Thought 9.39 6.92 4.24 0.43 11.34 18.66
+ Self-Reflection 17.72 13.51 0.00 0.00 0.00 1.69
+ Role-Playing 46.46 56.06 14.08 34.48 0.00 2.70
+ In-Context Learning 55.40 50.79 27.40 11.26 8.56 19.35

LLMs with prompt engineering vs. traditional ML models. Experiments on sampled MIMIC-III and MIMIC-IV test sets. Length-of-Stay uses Macro F1 (%), the other two tasks use F1 (%). In-Context Learning is the one strategy that clearly helps, and only on Length-of-Stay Prediction. Chain-of-Thought hurts almost everywhere, driving Llama3-70B to 0.00 F1 on Mortality and Readmission Prediction in both databases.

What we observe
  • The effectiveness of prompting strategies is generally very limited, though it varies by model and task. Apart from In-Context Learning boosting Llama3-8B, Llama3-70B and Meditron-70B on Length-of-Stay Prediction, the strategies bring no notable improvement and can degrade the original performance.
  • Traditional ML models such as XGBoost, SVM and RNN still outperform LLMs under every prompting strategy. The best case, Meditron-70B with In-Context Learning at 55.40 Macro F1 on MIMIC-III, stays well below SVM's 66.16.
Finding 3 RQ 3 · Can fine-tuned LLMs beat traditional ML models?
Fine-tuning can clearly enhance LLMs' performance on certain clinical prediction tasks but most fine-tuned LLMs still cannot beat traditional ML models yet.

Four LLMs are fine-tuned with two LoRA variants, one applied to all layers and one to the last layer only. Both use exactly the same training/validation/test split as the traditional ML models, run for 20 epochs, and select the checkpoint on the validation set.

EvidenceLoRA fine-tuning against traditional ML models
Bar charts comparing no fine-tuning, LoRA (Full) and LoRA (Last Layer) for Llama3-8B, Gemma2-9B, Vicuna-v1.5-7B and Mistral-v0.3-7B across three tasks and two databases, against dashed lines for Transformer, XGBoost, RNN and SVM
Fine-tuned LLMs vs. traditional ML models. Length-of-Stay uses Macro F1 (%), the others F1 (%). Dashed red lines mark Transformer, XGBoost, RNN and SVM. Fine-tuning produces large gains on Length-of-Stay, where Mistral-v0.3-7B goes from 11.20 to 53.89 on MIMIC-III. Almost every bar still falls short of the dashed lines.
What we observe
  • Fine-tuning clearly improves Length-of-Stay and Mortality Prediction, with LoRA (Full) giving a more evident gain than LoRA (Last Layer) on Length-of-Stay. Readmission Prediction shows no improvement at all.
  • Most fine-tuned LLMs still cannot surpass XGBoost, SVM, RNN and Transformer. Gemma2-9B with LoRA (Full) does beat three traditional ML models on Mortality Prediction in MIMIC-IV, but the same recipe fails to reproduce that gain on MIMIC-III.

Discussion

Implications and Hypothesis


High scores on medical licensing exams and question-answering benchmarks such as MedQA and MedMCQA suggest that LLMs carry clinician-level general medical knowledge. Whether that knowledge supports clinical reasoning and decision-making in real-world, complex scenarios has remained largely unexplored. By benchmarking predictive modeling head-to-head against traditional ML models, ClinicalBench sheds light on a potential deficiency.

We call for caution when practitioners adopt LLMs in clinical applications. ClinicalBench can be used to bridge the gap between LLM development for healthcare and real-world clinical practice.

Our Contributions


  • We built a new benchmark ClinicalBench, embracing 14 general-purpose LLMs, 8 medical LLMs, 11 traditional ML models, three tasks, and two databases, which made the first attempt to compare the clinical prediction capacities of LLMs and traditional ML models in a head-to-head way.
  • We discover that both general-purpose and medical LLMs, even with different model sizes, prompting or fine-tuning strategies, still cannot beat traditional ML models in clinical prediction yet.
  • Our findings demonstrate the potential deficiency of both general-purpose and medical LLMs in real-world clinical reasoning and decision-making, which could have almost clinician-level performance in medical licensing exams and clinical case challenges. We call for caution when adopting LLMs in practical clinical applications. ClinicalBench could be leveraged to bridge the gap between the development of LLMs for healthcare and real-world clinical practice.
  • We open-source our code and evaluation results for lasting assessment of both general-purpose and medical models, and to inspire more effective clinical prediction methods.

Limitations and Future Work


Limitations

  • ICD codes are administrative artifacts. Optimized for billing rather than clinical accuracy, they can under-code chronic conditions, over-code acute diagnoses, and vary across institutions and time. This affects all benchmarked methods equally and reflects standard practice in clinical prediction research.
  • A single medical center. Both MIMIC databases come from Beth Israel Deaconess Medical Center, and their ICU-centric population skews toward higher acuity than community hospitals, outpatient settings, or international contexts.
  • Standardized code descriptions. Our code-to-text conversion may differ from how clinicians naturally document in free-text notes, which may underestimate LLMs on authentic clinical narratives.

Future Directions

  • Incorporate multi-center databases to assess generalizability.
  • Evaluate on raw clinical notes rather than code-derived text.
  • Examine whether knowledge graph augmentation can bridge the gap between LLMs and traditional models.
  • Explore task-specific fine-tuning on clinical prediction objectives rather than general medical knowledge.

We acknowledge the importance of data privacy in clinical research and followed the data use guidelines of the MIMIC datasets.



Appendix

More Details and Results


Prompt Examples 3 tasks × 5 strategies, each with outputs from four LLMs

The exact prompt fed to every LLM, and what four representative models answered. Length-of-Stay expects 1/2/3. Mortality and Readmission expect 0/1. Chain-of-Thought and Self-Reflection additionally ask for reasoning, which is where several models drift off-task.

Prompt: Length-of-Stay Prediction, Directly Prompting
Patient information:
Age: 62
Gender: male
Conditions: Streptococcal septicemia, Pressure ulcer, lower back, Acute osteomyelitis, pelvic region and thigh, Unknown Diagnosis, Cocaine abuse, unspecified, Acute kidney failure, unspecified, Hemorrhage complicating a procedure, Hyperpotassemia, Chronic osteomyelitis, pelvic region and thigh, Sepsis, Paraplegia, Unspecified essential hypertension, Colostomy status, Other artificial opening of urinary tract status, Anemia of other chronic disease, Other iatrogenic hypotension, Candidiasis of other urogenital sites, Late effects of injury purposely inflicted by other person
Procedures: Local excision of lesion or tissue of bone, other bones, Local excision of lesion or tissue of bone, femur, Nonexcisional debridement of wound, infection or burn, Venous catheterization, not elsewhere classified, Transfusion of packed cells
Using Drugs: anesthetics, general, i.v. solution additives, other antibacterials, anterior pituitary lobe hormones and analogues, antithrombotic agents, other analgesics and antipyretics, drugs for constipation, drugs for peptic ulcer and gastro-oesophageal reflux disease (gord), antiseptics and disinfectants, antiinfectives and antiseptics, excl. combinations with corticosteroids, other nutrients, opioids, other beta-lactam antibacterials, other mineral supplements, other diagnostic agents, antipsychotics, i.v. solutions, antivaricose therapy, beta-lactam antibacterials, penicillins
Predict the number of weeks of stay in hospital.
Answer 1 if no more than one week,
Answer 2 if more than one week but not more than two weeks,
Answer 3 if more than two weeks.
Answer with only the number. Answer:
Model outputs
Llama3-8B
3
Gemma2-9B
3
Meditron-7B
3
Internist-7B
3
Fine-Tuning Data Construction instruction / input / output triples

How each clinical prediction task is turned into supervised fine-tuning data for the LoRA experiments.


Length-of-Stay Prediction
Instruction
Given the patient information, predict the number of weeks of stay in hospital.
Answer 1 if no more than one week,
Answer 2 if more than one week but not more than two weeks,
Answer 3 if more than two weeks.
Answer with only the number
Input
Patient information:
Age: 78
Gender: male
Conditions: Coronary atherosclerosis of native coronary artery, Chronic total occlusion of coronary artery, Malignant neoplasm of rectum, Personal history of antineoplastic chemotherapy, Person boarding or alighting a pedal cycle injured in collision with railway train or railway vehicle
Procedures: Percutaneous transluminal coronary angioplasty [PTCA], Insertion of non-drug-eluting coronary artery stent(s), Left heart cardiac catheterization, Coronary arteriography using two catheters, Insertion of two vascular stents, Procedure on two vessels
Using Drugs: other analgesics and antipyretics, lipid modifying agents, plain, antithrombotic agents, i.v. solution additives, antidepressants, potassium, viral vaccines, vasodilators used in cardiac diseases, belladonna and derivatives, plain
Answer:
Output
1
Mortality Prediction
Instruction
Given the patient information, predict the mortality of the patient.
Answer 1 if the patient will die, answer 0 otherwise.
Answer with only the number
Input
Patient information:
Age: 64
Gender: female
Conditions: Osteoarthrosis, localized, not specified whether primary or secondary, lower leg, Other and unspecified angina pectoris, Unspecified essential hypertension, Mitral valve disorders, Psoriatic arthropathy
Procedures: Total knee replacement
Using Drugs: selective calcium channel blockers with mainly vascular effects, beta blocking agents, lipid modifying agents, plain, vitamin a and d, incl. combinations of the two, antihistamines for systemic use, other nutrients, macrolides, lincosamides and streptogramins, anticholinergic agents, drugs for peptic ulcer and gastro-oesophageal reflux disease (gord), drugs for constipation, opioids, antidepressants, other analgesics and antipyretics, i.v. solutions, antivaricose therapy, antiemetics and antinauseants, antiinflammatory and antirheumatic products, non-steroids, antacids, calcium, angiotensin ii receptor blockers (arbs), plain, antithrombotic agents
Will the patient die because of the above situation?
Answer:
Output
0
Readmission Prediction
Instruction
Given the patient information, predict the readmission of the patient.
Answer 1 if the patient will be readmitted to the hospital within two weeks, answer 0 otherwise.
Answer with only the number
Input
Patient information:
Age: 91
Gender: female
Conditions: Open wound of forehead, without mention of complication, Hyposmolality and/or hyponatremia, Open wound of knee, leg [except thigh], and ankle, without mention of complication, Contusion of thigh, Contusion of scapular region, Open wound of scalp, without mention of complication, Unspecified essential hypertension, Insomnia, unspecified, Unspecified deficiency anemia, Acquired absence of both cervix and uterus, Acquired absence of intestine (large) (small), Personal history of malignant melanoma of skin, Fall from other slipping, tripping, or stumbling
Procedures: Closure of skin and subcutaneous tissue of other sites
Using Drugs: selective calcium channel blockers with mainly vascular effects, vitamin a and d, incl. combinations of the two, antidepressants, other analgesics and antipyretics, beta blocking agents, drugs for constipation, antithrombotic agents, antacids, calcium, i.v. solution additives
Will the patient be readmitted to the hospital within two weeks?
Answer:
Output
0
Traditional ML Models on Different Training-Set Scales 5%, 10%, 20%, 40% of the training set

Evidence for insight ③. Even trained on a fraction of the data, traditional ML models stay stable and still beat every LLM. Training subsets are drawn by stratified sampling from the original training set, and validation and test sets are unchanged.


Method Length-of-Stay Prediction Mortality Prediction Readmission Prediction
Macro F1 (%)AUROC (%) F1 (%)AUROC (%) F1 (%)AUROC (%)
Majority 23.37(23.37, 23.37) 50.00(50.00, 50.00) 0.00(0.00, 0.00) 50.00(50.00, 50.00) 0.00(0.00, 0.00) 50.00(50.00, 50.00)
Minority 10.72(10.72, 10.72) 50.00(50.00, 50.00) 20.88(20.88, 20.88) 50.00(50.00, 50.00) 13.64(13.64, 13.64) 50.00(50.00, 50.00)
Traditional ML Models with 40% of Original Training Set from MIMIC-III
XGBoost 63.99(63.84, 64.14) 91.53(91.45, 91.61) 62.16(62.03, 62.29) 94.67(94.63, 94.71) 17.69(17.59, 17.79) 64.23(64.04, 64.42)
Logistic Regression 62.44(62.28, 62.60) 90.61(90.47, 90.75) 60.24(60.14, 60.34) 92.91(92.88, 92.94) 18.32(18.10, 18.54) 66.06(65.68, 66.44)
Decision Tree 56.14(55.99, 56.29) 83.16(82.98, 83.34) 48.41(48.20, 48.62) 81.17(80.89, 81.45) 14.95(14.86, 15.04) 53.61(53.31, 53.91)
Random Forest 61.06(60.96, 61.16) 90.88(90.78, 90.98) 53.11(52.96, 53.26) 90.78(90.70, 90.86) 18.34(18.09, 18.59) 66.40(65.95, 66.85)
AdaBoost 62.54(62.34, 62.74) 89.38(89.09, 89.67) 57.04(56.76, 57.32) 90.94(90.78, 91.10) 16.26(16.09, 16.43) 59.77(59.27, 60.27)
SVM 65.05(64.90, 65.20) 91.71(91.63, 91.79) 60.51(60.30, 60.72) 94.04(93.99, 94.09) 17.97(17.70, 18.24) 66.84(66.48, 67.20)
Naive Bayes 45.57(45.48, 45.66) 62.48(62.28, 62.68) 24.86(24.81, 24.91) 59.10(59.02, 59.18) 16.63(16.51, 16.75) 58.05(57.81, 58.29)
KNN 45.34(45.24, 45.44) 74.43(74.28, 74.58) 42.17(41.90, 42.44) 83.75(83.59, 83.91) 15.25(15.14, 15.36) 59.80(59.38, 60.22)
MLP 59.92(59.75, 60.09) 87.99(87.81, 88.17) 57.00(56.85, 57.15) 90.77(90.70, 90.84) 18.25(18.07, 18.43) 65.71(65.38, 66.04)
Transformer 58.73(58.45, 59.01) 87.22(87.02, 87.42) 56.05(55.81, 56.29) 91.74(91.71, 91.77) 18.21(18.04, 18.38) 65.49(65.19, 65.79)
RNN 61.07(60.88, 61.26) 88.08(87.94, 88.22) 58.77(58.60, 58.94) 90.08(89.96, 90.20) 19.02(18.86, 19.18) 67.38(67.11, 67.65)
Traditional ML Models with 20% of Original Training Set from MIMIC-III
XGBoost 61.95(61.81, 62.09) 90.89(90.78, 91.00) 57.37(57.07, 57.67) 92.86(92.78, 92.94) 16.55(16.35, 16.75) 61.33(60.79, 61.87)
Logistic Regression 60.78(60.55, 61.01) 89.71(89.52, 89.90) 56.55(56.24, 56.86) 91.57(91.47, 91.67) 16.69(16.45, 16.93) 63.59(63.19, 63.99)
Decision Tree 54.8(54.64, 54.96) 79.85(79.65, 80.05) 44.23(43.66, 44.80) 77.20(76.99, 77.41) 15.01(14.86, 15.16) 56.25(55.94, 56.56)
Random Forest 60.48(60.28, 60.68) 90.59(90.48, 90.70) 52.00(51.85, 52.15) 90.15(90.05, 90.25) 16.90(16.77, 17.03) 63.09(62.93, 63.25)
AdaBoost 60.51(60.32, 60.70) 87.97(87.76, 88.18) 53.76(53.60, 53.92) 88.92(88.77, 89.07) 15.75(15.50, 16.00) 58.43(58.01, 58.85)
SVM 63.05(62.92, 63.18) 91.10(91.02, 91.18) 57.35(57.01, 57.69) 92.72(92.64, 92.80) 17.41(17.20, 17.62) 64.88(64.58, 65.18)
Naive Bayes 43.82(43.77, 43.87) 62.88(62.70, 63.06) 25.63(25.57, 25.69) 60.44(60.34, 60.54) 16.71(16.54, 16.88) 58.02(57.67, 58.37)
KNN 44.49(44.33, 44.65) 74.31(74.14, 74.48) 42.39(42.21, 42.57) 82.91(82.77, 83.05) 15.04(14.94, 15.14) 58.93(58.47, 59.39)
MLP 58.66(58.41, 58.91) 87.28(87.05, 87.51) 53.54(53.24, 53.84) 89.98(89.84, 90.12) 17.08(16.80, 17.36) 62.81(62.39, 63.23)
Transformer 58.69(58.44, 58.94) 88.18(87.96, 88.40) 53.02(52.84, 53.20) 90.74(90.56, 90.92) 17.04(16.88, 17.20) 61.73(61.37, 62.09)
RNN 61.16(60.89, 61.43) 88.74(88.51, 88.97) 55.77(55.36, 56.18) 89.13(88.89, 89.37) 17.85(17.66, 18.04) 64.03(63.78, 64.28)
Majority 23.37(23.37, 23.37) 50.00(50.00, 50.00) 0.00(0.00, 0.00) 50.00(50.00, 50.00) 0.00(0.00, 0.00) 50.00(50.00, 50.00)
Minority 10.72(10.72, 10.72) 50.00(50.00, 50.00) 20.88(20.88, 20.88) 50.00(50.00, 50.00) 13.64(13.64, 13.64) 50.00(50.00, 50.00)
Traditional ML Models with 10% of Original Training Set from MIMIC-III
XGBoost 61.35(61.18, 61.52) 90.16(90.05, 90.27) 51.62(51.37, 51.87) 89.93(89.82, 90.04) 17.05(16.81, 17.29) 61.43(60.79, 62.07)
Logistic Regression 60.79(60.64, 60.94) 90.25(90.10, 90.40) 53.92(53.64, 54.20) 90.49(90.39, 90.59) 17.71(17.54, 17.88) 62.21(61.85, 62.57)
Decision Tree 53.07(52.98, 53.16) 77.11(76.77, 77.45) 41.43(41.16, 41.70) 74.65(74.43, 74.87) 15.32(15.17, 15.47) 55.75(55.36, 56.14)
Random Forest 60.39(60.25, 60.53) 90.61(90.54, 90.68) 50.00(49.75, 50.25) 88.72(88.63, 88.81) 17.47(17.35, 17.59) 63.51(63.32, 63.70)
AdaBoost 60.3(60.23, 60.37) 87.55(87.40, 87.70) 50.07(49.63, 50.51) 85.78(85.61, 85.95) 16.03(15.80, 16.26) 57.48(57.07, 57.89)
SVM 62.33(62.2, 62.46) 90.73(90.66, 90.8) 52.51(52.17, 52.85) 90.97(90.86, 91.08) 17.01(16.81, 17.21) 52.28(50.47, 54.09)
Naive Bayes 41.54(41.39, 41.69) 63.82(63.65, 63.99) 26.48(26.32, 26.64) 62.18(61.81, 62.55) 14.87(14.79, 14.95) 54.25(54.11, 54.39)
KNN 42.37(42.23, 42.51) 73.74(73.49, 73.99) 41.26(41.13, 41.39) 82.37(82.31, 82.43) 14.30(14.23, 14.37) 57.96(57.27, 58.65)
MLP 58.65(58.39, 58.91) 87.51(87.32, 87.70) 51.55(51.17, 51.93) 89.57(89.48, 89.66) 15.95(15.83, 16.07) 59.23(59.00, 59.46)
Transformer 58.94(58.63, 59.25) 89.43(89.25, 89.61) 50.19(49.81, 50.57) 89.43(89.31, 89.55) 15.76(15.52, 16.00) 58.34(57.91, 58.77)
RNN 61.81(61.54, 62.08) 89.20(89.04, 89.36) 54.62(54.32, 54.92) 89.00(88.88, 89.12) 13.97(13.55, 14.39) 61.82(61.65, 61.99)
Traditional ML Models with 5% of Original Training Set from MIMIC-III
XGBoost 59.21(59.00, 59.42) 89.16(89.00, 89.32) 48.60(48.39, 48.81) 87.67(87.58, 87.76) 16.40(16.18, 16.62) 58.90(58.49, 59.31)
Logistic Regression 59.90(59.74, 60.06) 89.92(89.74, 90.10) 52.87(52.63, 53.11) 89.50(89.36, 89.64) 15.51(15.37, 15.65) 58.96(58.50, 59.42)
Decision Tree 52.35(52.17, 52.53) 76.17(75.98, 76.36) 39.02(38.44, 39.60) 70.48(69.96, 71.00) 15.90(15.77, 16.03) 56.24(56.01, 56.47)
Random Forest 60.89(60.74, 61.04) 90.34(90.26, 90.42) 52.10(51.83, 52.37) 89.15(88.99, 89.31) 15.75(15.58, 15.92) 60.29(59.84, 60.74)
AdaBoost 56.63(56.35, 56.91) 86.61(86.43, 86.79) 45.37(45.06, 45.68) 83.13(82.77, 83.49) 15.41(15.33, 15.49) 59.09(58.89, 59.29)
SVM 61.33(61.20, 61.46) 90.10(90.00, 90.20) 52.58(52.24, 52.92) 90.04(89.93, 90.15) 15.95(15.73, 16.17) 49.55(47.98, 51.12)
Naive Bayes 42.91(42.71, 43.11) 66.12(65.90, 66.34) 29.89(29.72, 30.06) 66.44(66.25, 66.63) 14.06(13.82, 14.30) 53.30(53.02, 53.58)
KNN 39.98(39.91, 40.05) 71.55(71.39, 71.71) 40.70(40.43, 40.97) 83.04(82.93, 83.15) 14.26(14.23, 14.29) 55.74(55.13, 56.35)
MLP 57.97(57.79, 58.15) 87.22(87.03, 87.41) 50.27(50.04, 50.50) 87.92(87.77, 88.07) 14.91(14.82, 15.00) 57.36(57.01, 57.71)
Transformer 58.23(57.95, 58.51) 88.73(88.54, 88.92) 48.67(48.51, 48.83) 87.88(87.68, 88.08) 14.04(13.72, 14.36) 54.06(53.55, 54.57)
RNN 60.94(60.74, 61.14) 88.19(88.03, 88.35) 52.84(52.61, 53.07) 88.04(87.85, 88.23) 11.92(11.21, 12.63) 59.50(59.16, 59.84)
Majority 29.56(29.56, 29.56) 50.0(50.0, 50.0) 0.0(0.0, 0.0) 50.0(50.0, 50.0) 0.0(0.0, 0.0) 50.0(50.0, 50.0)
Minority 4.56(4.56, 4.56) 50.0(50.0, 50.0) 6.7(6.7, 6.7) 50.0(50.0, 50.0) 24.92(24.92, 24.92) 50.0(50.0, 50.0)
Traditional ML Models with 40% of Original Training Set from MIMIC-IV
XGBoost 61.81(61.70, 61.92) 94.51(94.46, 94.56) 44.06(43.66, 44.46) 97.36(97.31, 97.41) 26.87(26.77, 26.97) 58.86(58.69, 59.03)
Logistic Regression 60.60(60.41, 60.79) 93.64(93.58, 93.70) 47.10(46.86, 47.34) 97.63(97.59, 97.67) 26.24(26.15, 26.33) 58.01(57.84, 58.18)
Decision Tree 53.07(52.76, 53.38) 86.57(86.50, 86.64) 30.97(30.42, 31.52) 85.31(85.00, 85.62) 25.33(25.05, 25.61) 54.97(54.84, 55.10)
Random Forest 57.86(57.78, 57.94) 92.06(91.97, 92.15) 34.24(33.77, 34.71) 95.18(95.04, 95.32) 27.08(27.00, 27.16) 59.52(59.35, 59.69)
AdaBoost 60.64(60.53, 60.75) 92.50(92.45, 92.55) 43.14(42.75, 43.53) 95.29(95.11, 95.47) 26.17(26.11, 26.23) 56.45(56.29, 56.61)
SVM 64.50(64.39, 64.61) 95.48(95.42, 95.54) 42.79(42.41, 43.17) 97.51(97.47, 97.55) 28.32(28.22, 28.42) 60.85(60.70, 61.00)
Naive Bayes 42.47(42.37, 42.57) 70.00(69.84, 70.16) 16.13(15.99, 16.27) 77.47(77.36, 77.58) 25.25(25.14, 25.36) 52.79(52.58, 53.00)
KNN 43.54(43.31, 43.77) 71.62(71.35, 71.89) 38.66(38.15, 39.17) 90.02(89.83, 90.21) 23.11(22.78, 23.44) 53.63(53.35, 53.91)
MLP 58.70(58.62, 58.78) 90.83(90.75, 90.91) 42.41(42.14, 42.68) 97.17(97.14, 97.20) 25.81(25.65, 25.97) 56.94(56.76, 57.12)
Transformer 57.66(57.44, 57.88) 91.56(91.47, 91.65) 40.94(40.48, 41.40) 97.38(97.33, 97.43) 25.33(25.27, 25.39) 56.36(56.27, 56.45)
RNN 61.63(61.53, 61.73) 91.74(91.69, 91.79) 46.56(46.37, 46.75) 96.89(96.85, 96.93) 25.56(25.47, 25.65) 56.74(56.60, 56.88)
Traditional ML Models with 20% of Original Training Set from MIMIC-IV
XGBoost 60.05(59.97, 60.13) 93.49(93.41, 93.57) 38.80(38.51, 39.09) 96.21(96.15, 96.27) 25.16(25.05, 25.27) 56.43(56.21, 56.65)
Logistic Regression 59.51(59.31, 59.71) 92.87(92.71, 93.03) 42.81(42.55, 43.07) 96.91(96.84, 96.98) 25.27(25.16, 25.38) 56.00(55.89, 56.11)
Decision Tree 52.15(51.83, 52.47) 84.08(83.91, 84.25) 27.27(26.49, 28.05) 83.42(83.04, 83.80) 21.39(20.78, 22.00) 52.87(52.66, 53.08)
Random Forest 57.70(57.55, 57.85) 91.85(91.75, 91.95) 34.77(34.30, 35.24) 95.14(95.01, 95.27) 26.55(26.46, 26.64) 58.08(57.85, 58.31)
AdaBoost 57.76(57.58, 57.94) 91.66(91.58, 91.74) 38.77(38.60, 38.94) 94.49(94.27, 94.71) 24.88(24.80, 24.96) 54.71(54.62, 54.80)
SVM 63.14(63.06, 63.22) 94.88(94.82, 94.94) 39.96(39.57, 40.35) 96.53(96.46, 96.60) 26.62(26.45, 26.79) 58.14(57.93, 58.35)
Naive Bayes 39.13(39.00, 39.26) 70.63(70.41, 70.85) 14.53(14.40, 14.66) 75.68(75.44, 75.92) 24.66(24.46, 24.86) 53.69(53.48, 53.90)
KNN 44.05(43.75, 44.35) 72.61(72.43, 72.79) 38.87(38.39, 39.35) 87.61(87.30, 87.92) 22.96(22.78, 23.14) 53.41(53.23, 53.59)
MLP 57.34(57.16, 57.52) 89.89(89.76, 90.02) 39.84(39.51, 40.17) 96.59(96.53, 96.65) 24.37(24.29, 24.45) 54.71(54.58, 54.84)
Transformer 57.39(57.19, 57.59) 91.72(91.57, 91.87) 35.91(35.18, 36.64) 96.46(96.38, 96.54) 24.35(24.24, 24.46) 54.57(54.43, 54.71)
RNN 60.27(60.07, 60.47) 90.79(90.62, 90.96) 42.13(41.87, 42.39) 96.20(96.13, 96.27) 24.77(24.67, 24.87) 55.31(55.23, 55.39)
Majority 29.56(29.56, 29.56) 50.0(50.0, 50.0) 0.0(0.0, 0.0) 50.0(50.0, 50.0) 0.0(0.0, 0.0) 50.0(50.0, 50.0)
Minority 4.56(4.56, 4.56) 50.0(50.0, 50.0) 6.7(6.7, 6.7) 50.0(50.0, 50.0) 24.92(24.92, 24.92) 50.0(50.0, 50.0)
Traditional ML Models with 10% of Original Training Set from MIMIC-IV
XGBoost 56.22(56.14, 56.30) 91.79(91.69, 91.89) 33.59(33.07, 34.11) 94.67(94.53, 94.81) 24.84(24.69, 24.99) 54.70(54.42, 54.98)
Logistic Regression 57.96(57.80, 58.12) 92.13(92.00, 92.26) 40.20(39.65, 40.75) 96.25(96.18, 96.32) 24.49(24.36, 24.62) 54.82(54.58, 55.06)
Decision Tree 49.48(49.12, 49.84) 80.08(79.75, 80.41) 26.82(26.23, 27.41) 80.25(79.76, 80.74) 24.23(23.78, 24.68) 53.16(52.91, 53.41)
Random Forest 57.15(56.97, 57.33) 91.66(91.54, 91.78) 35.29(34.67, 35.91) 95.16(94.97, 95.35) 25.42(25.20, 25.64) 56.09(55.81, 56.37)
AdaBoost 54.47(54.31, 54.63) 90.96(90.80, 91.12) 39.87(39.48, 40.26) 95.36(95.29, 95.43) 23.13(22.95, 23.31) 52.50(52.33, 52.67)
SVM 62.02(61.87, 62.17) 94.09(94.01, 94.17) 36.53(36.12, 36.94) 95.80(95.66, 95.94) 24.72(24.51, 24.93) 50.87(50.10, 51.64)
Naive Bayes 36.28(36.20, 36.36) 71.22(71.14, 71.30) 13.66(13.58, 13.74) 75.26(75.19, 75.33) 24.06(23.90, 24.22) 53.51(53.39, 53.63)
KNN 40.62(40.34, 40.90) 70.57(70.17, 70.97) 34.72(34.03, 35.41) 85.06(84.81, 85.31) 22.31(22.20, 22.42) 52.84(52.62, 53.06)
MLP 56.07(55.96, 56.18) 88.97(88.87, 89.07) 36.12(35.78, 36.46) 95.75(95.71, 95.79) 23.86(23.74, 23.98) 53.71(53.54, 53.88)
Transformer 55.37(55.24, 55.50) 89.94(89.82, 90.06) 30.94(30.54, 31.34) 95.21(95.12, 95.30) 24.18(24.05, 24.31) 53.91(53.75, 54.07)
RNN 59.04(58.94, 59.14) 89.39(89.27, 89.51) 39.13(38.77, 39.49) 95.64(95.58, 95.70) 24.37(24.26, 24.48) 54.44(54.26, 54.62)
Traditional ML Models with 5% of Original Training Set from MIMIC-IV
XGBoost 54.47(54.38, 54.56) 89.99(89.76, 90.22) 25.04(24.52, 25.56) 90.50(90.33, 90.67) 24.39(24.25, 24.53) 53.80(53.53, 54.07)
Logistic Regression 57.20(57.03, 57.37) 91.51(91.35, 91.67) 34.45(33.87, 35.03) 94.75(94.62, 94.88) 23.96(23.78, 24.14) 53.57(53.22, 53.92)
Decision Tree 49.15(48.75, 49.55) 75.08(73.98, 76.18) 22.11(21.63, 22.59) 78.31(77.88, 78.74) 22.53(21.91, 23.15) 52.28(52.01, 52.55)
Random Forest 57.40(57.28, 57.52) 91.53(91.35, 91.71) 34.35(33.68, 35.02) 94.07(93.88, 94.26) 24.94(24.76, 25.12) 54.40(53.97, 54.83)
AdaBoost 50.14(49.93, 50.35) 90.59(90.47, 90.71) 37.37(36.54, 38.20) 92.59(92.34, 92.84) 23.44(23.28, 23.60) 52.27(51.99, 52.55)
SVM 60.63(60.36, 60.90) 93.00(92.86, 93.14) 32.92(32.52, 33.32) 93.75(93.61, 93.89) 24.75(24.52, 24.98) 50.00(49.23, 50.77)
Naive Bayes 34.42(34.32, 34.52) 70.08(69.74, 70.42) 12.74(12.58, 12.90) 73.99(73.69, 74.29) 23.14(23.04, 23.24) 52.19(52.07, 52.31)
KNN 38.72(38.33, 39.11) 67.40(67.07, 67.73) 24.09(23.40, 24.78) 79.08(78.66, 79.50) 22.13(21.89, 22.37) 52.32(52.13, 52.51)
MLP 54.25(54.10, 54.40) 86.36(86.17, 86.55) 32.24(31.75, 32.73) 94.16(94.05, 94.27) 23.68(23.61, 23.75) 53.13(52.86, 53.40)
Transformer 54.50(54.35, 54.65) 88.37(88.09, 88.65) 22.59(22.04, 23.14) 91.43(91.19, 91.67) 23.99(23.87, 24.11) 52.83(52.58, 53.08)
RNN 57.88(57.70, 58.06) 87.51(87.14, 87.88) 17.32(16.98, 17.66) 92.38(92.25, 92.51) 24.09(23.96, 24.22) 53.77(53.48, 54.06)

Average over five runs. Brackets give the 95% confidence interval. Majority / Minority are repeated within each block for reference.

Full Confusion Matrices 26 methods × 3 tasks × 2 databases

Each grid shows all 11 traditional ML models, then 9 general-purpose LLMs, then 6 medical LLMs, in the order used in the main results table.

Confusion matrices for all 26 benchmarked methods
Fine-Tuning Loss Curves 4 LLMs × 3 tasks, both LoRA variants

Evaluation loss over training steps, showing that the LLMs mostly converge within the 20 training epochs used in the fine-tuning experiments. The best checkpoint is selected on the validation set.

Fine-tuning evaluation loss curves
Reproducibility & Model Checkpoints hardware, decoding settings, all 22 LLMs

BibTeX

@inproceedings{chen2026clinicalbench,
  title     = {ClinicalBench: Can LLMs Beat Traditional ML Models in Clinical Prediction?},
  author    = {Chen, Canyu and Yu, Jian and Chen, Shan and Liu, Che and Wan, Zhongwei
               and Zhou, Shuang and Luo, Yuan and Zhang, Rui and Bitterman, Danielle S.
               and Wang, Fei and Shu, Kai},
  booktitle = {Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery
               and Data Mining (KDD '26)},
  year      = {2026}
}