A Systematic Cross-Family Comparison of Fine-Tuning Open-Weight LLMs for Traditional Chinese Medicine: Capacity Scaling, Family Effects, and Teachability

— An empirical study of 8 open-weight base models (7B–72B) fine-tuned for Chinese medical licensing examinations under identical data and identical recipe

v1.3 (2026-07-27, adds 14-language multilingual evaluation + safety/typo results for the 3 new bases) · AIMate Smart Clinic Research Team · Technical report

Abstract

Aiming at a locally deployable, TCM-specialized LLM for primary-care clinics, we systematically trained and evaluated 8 open-weight base models (7 completed full training and evaluation; 1 aborted) spanning three families — Qwen (R1-Distill / Qwen2.5 / Qwen3), GLM-4, and Llama — across multiple generations and a 7B–72B capacity range, under identical training data (270,208 samples) and an identical fine-tuning recipe. We ask: (1) Does capacity scaling yield consistent gains? (2) Does switching model family/generation add further gains? (3) How does a local open-weight specialist compare with a frontier web-connected model?

Key findings: ① The capacity ladder is monotonic — across R1-Distill-Qwen 7B→14B→32B→72B, TCM MCQ accuracy rises 36.0%→71.0%→76.3%→85.3% and general medicine 43.7%→73.7%→83.8%→90.3%; every model's base→tuned gain is significant by paired McNemar test (p≈0). ② Family/generation effects are real — at 14B, native Qwen3 beats R1-Distill-Qwen-14B on both axes (TCM 73.5% vs 71.0%; general 80.0% vs 73.7%); DeepSeek-R1-0528-Qwen3-8B shows the strongest “teachability” in the study (base 4.2% → tuned 65.7%, Δ+61.5pp). ③ The local 72B ties the web-connected frontier — the fully offline Qwen2.5-72B fine-tune is statistically tied with the web-connected frontier v4-pro on the n=811/801 large held-out set (TCM 85.6% vs 86.8%, p=0.47; general 89.8% vs 87.6%, p=0.18). Safety evaluation reveals a capacity phase transition: prescription-review specificity jumps from ~0.03 to 0.73 at 32B, with hallucination rate dropping 9×. All raw evaluation data, significance statistics and training provenance are released with this paper.

Keywords: TCM large language model; parameter-efficient fine-tuning (QLoRA); model capacity; cross-family comparison; decontaminated evaluation; McNemar test

1. Introduction

Intelligent assistance for primary-care TCM clinics (speech transcription, medical-record generation, prescription safety review) imposes three hard constraints on the language model: specialization (TCM syndrome-differentiation knowledge and licensing-exam-level competence), locality (patient data must not leave the clinic; the model must run quantized on consumer GPUs), and trustworthiness (safety review that neither over-flags nor under-flags). These constraints dictate our technical route: specialization via fine-tuning open-weight bases on domain data, rather than relying on cloud general-purpose APIs.

Within this route, the key engineering and scientific question is: which base model? Capacity (7B/14B/32B/72B), family (Qwen/GLM/Llama), generation (Qwen2.5/Qwen3), and reasoning lineage (R1-distilled vs. native) interweave; community wisdom mostly comes from general English benchmarks and may not transfer to Chinese medical specialization. Anecdotal conclusions (“bigger is always better” or “small is enough”) are unreliable — only controlled variables — same data, same recipe, same evaluation, swapping one base at a time — yield decision-grade evidence.

This paper is the complete record of that controlled experiment. Our contributions:

1. A reproducible cross-base comparison table: 8 bases (7 completed + 1 aborted) × one 270,208-sample training set × one QLoRA recipe × one large held-out set (n=1,612), with paired statistical significance throughout;
2. Three empirical conclusions: a monotonic capacity ladder, independent family/generation gains, and “teachability” that is negatively correlated with base scores;
3. A safety capacity-transition finding: prescription-review specificity jumps at 32B (0.03→0.73), directly informing the design of clinical review pipelines;
4. Complete data governance and honest boundaries: the held-out set is verified disjoint from training data through multiple independent audits; all non-comparable factors (library-compatibility discounts, early convergence stops) are disclosed.

2. Data

2.1 Training set

All runs use the same v3 training set of 270,208 samples from 8 sources (per-file SHA-256 prefixes verified identical across runs via run_manifest.json):

2.2 Evaluation set (held-out) and decontamination

The main evaluation is the CMB large held-out: stratified from CMB-train, near-duplicate decontaminated (384 rewritten duplicates discarded), then deleted from the training set (→ cmb_full_v2), yielding cmb_tcm (TCM) n=811 and cmb_medical (general medicine) n=801. The external benchmark CMExam was rejected as an evaluation set (19.2% verbatim overlap with CMB-train — not independent) and is used only as training material.

Decontamination was verified in three independent rounds: (1) a near-duplicate gate at split time (0/1,609 leaked); (2) char-8gram containment + exact-substring checks against all 8 training files — zero leakage; (3) a 2026-07-14 credibility re-audit with cross-source exact matching — residual contamination effect ≤0.12pp, negligible. The safety evaluation set was expanded to n=195 (30 clean standard formulas + 165 erroneous cases, all self-checked by check_prescription_safety and disjoint from the training review pool). The typo set has n=90.

2.3 Data compliance

All training data comes from public academic benchmarks and self-built knowledge bases — no real patient data (PII-free). The residual risk is uncleared per-question copyright of upstream exam questions (an industry-wide gray zone), which we disclose here.

3. Methods

3.1 Fine-tuning recipe

Local runs and the 32B cloud run use our in-house train_lora.py: QLoRA 4-bit + packing, epochs=1, LoRA rank=16 (alpha=32), lr=1e-4, cutoff 1024 (32B cloud run 2048). The decisive reason for our own stack over LLaMA-Factory is the health gate: after training, reasoning models are screened for degeneration (empty <think>, repetition collapse), with exit code 3 blocking bad models — we previously produced a completely wrong “32B≈14B” conclusion when LLaMA-Factory's deepseekr1 template broke the reasoning model on no-think data with no gate to catch it (retracted; see §6.4). The 72B model, infeasible on a single machine, was trained with AWS 4×g6e.2xlarge multi-node DDP (LLaMA-Factory + Liger + paged_adamw_8bit, template=qwen; the non-reasoning base is immune to that failure mode).

3.2 Evaluation protocol

All models share one harness: base 4-bit + LoRA adapter, batch 8, max_new_tokens=512 (GLM-4 excepted, see §7). Each model is evaluated base first, then tuned, on the same questions paired. Scoring is exact set equality (multi-select answers must match letter-for-letter, preventing lucky-guess inflation). The empty-answer rate is 0 throughout, ruling out “format not learned” artifacts — base→tuned gains are knowledge gains, not format gains.

3.3 Statistics

Base→tuned and cross-model paired comparisons use the McNemar test (same-question pairing; win/loss counts and p-values reported); comparisons against the frontier model (independent samples) use the two-proportion z-test; accuracies are reported with Wilson 95% CIs.

4. Experimental Setup: Training Overview

Notes: 72B had max_steps=5,266, but its loss entered a flat plateau from step ~130 (mean 0.084 / std 0.036, no downward trend); training was stopped at checkpoint-2400 (46%) on judged convergence, saving ~$25 with provably zero remaining gain. Llama-70B reached only 150/755 steps (0.06 epoch) in 3h40m; resuming would cost ~$45 with insufficient marginal value — aborted, checkpoint retained for future resume.

5. Results

5.1 Main results: MCQ large held-out (cmb_tcm n=811 / cmb_medical n=801)

All 7 models show McNemar p≈0 (significant) base→tuned gains. ⚠️ GLM-4, due to three incompatibilities between its remote code and transformers 4.57 forcing use_cache=False, could only be evaluated at max_new=128 (vs. 512 for others); its scores carry a library-compatibility discount (estimated 2–5pp underestimation) and are not strictly comparable.

5.2 Comparison with the web-connected frontier

v4-pro (web-connected, able to look up answers) on the same held-out, prompt and scoring: TCM 86.8%, general 87.6%. Two-proportion z-tests: TCM 72B 85.6% vs 86.8%, z=−0.72, p=0.47 (tie); general 72B 89.8% vs 87.6%, z=+1.34, p=0.18 (tie). A fully offline open-weight 72B specialist is in the same league as the web-connected frontier. An earlier small-sample (n=55) run once suggested “beating the frontier”; re-measuring on a 15× larger sample corrected this to a tie — the accurate statement is “same tier, drifting slightly with question distribution”.

5.3 Safety evaluation (prescription review, sensitivity/specificity)

Capacity transition: specificity jumps 0.03→0.73 from 14B to 32B — the 7B/14B/72B models all suffer “flag-everything” disease (specificity ≈ 0, flagging even clean formulas), first cured at the 32B tier; hallucination drops 0.374→0.041 (9×). The production review pipeline still uses the dedicated balanced 7b-safe (0.48/0.90, trained with rebalanced ratios; its specificity beats the frontier API's 0.10).

5.4 Typo correction (n=90)

5.5 Multilingual evaluation (14 languages, new in v1.3)

Method: same paper, different languages — one identical clean held-out (797/787) was machine-translated into 13 languages under a 105-term glossary constraint (answer letters unchanged, scoring logic untouched), then evaluated per language on the same harness; zh uses the original clean set as the anchor, and Δ is each language's gap to the zh-clean baseline.

Main matrix (tuned accuracy %, with Δ vs zh):

Five conclusions:

1. TCM knowledge is strongly Chinese-bound — TCM scores degrade in every language (−10 to −45pp) and far more than general medicine; serving other languages requires multilingual TCM training data;
2. general medicine is resilient — English is nearly lossless (14b-v3 even +0.4pp, within noise);
3. qwen3-14b beats 14b-v3 in every language (14/14, both axes) — the flagship choice holds multilingually;
4. low-resource languages (kk/uz) collapse (14b-v3 kk TCM only 25.6%) — these regions must currently fall back to English;
5. RTL (ar/fa) carries no extra penalty.

6. Discussion

6.5 Deployment route

• Knowledge/generation workhorse: distill 72B→14B/7B (compress CMB specialization into clinic-runnable quantized small models);
• When syndrome-differentiation CoT is needed: 32B-full-lora (reasoning preserved);
• Safety review: the dedicated balanced 7b-safe (not a knowledge model — knowledge models “flag everything” on the safety axis, specificity 0.10, unusable for review).

7. Limitations

1. GLM-4 evaluation discount: library incompatibility forced use_cache=False (O(n²) generation), with evaluation at max_new=128; scores may be underestimated by 2–5pp. Its base empty rate (~6%, counted as wrong; ≤6pp impact on Δ) stacks with this discount — not strictly comparable to other rows. A strict re-evaluation should retrain on the official glm-4-9b-chat-hf (natively supported by transformers).
2. Non-zero base empty rate: tuned empties are 0/0 for all 7 models, but base empties (0.1%–6.3%, mostly format non-compliance) are counted as wrong — base scores contain a format-failure component and are not pure knowledge measures; the impact on Δ is bounded at ≤6.3pp and changes no significance conclusion (per-model quantification in analysis_reports/eval_coverage_audit_2026-07-22.md).
3. The 72B is a converged checkpoint (epoch 0.46, within the flat loss plateau), not a full run; it matches the full-run scores of its series within noise, but strictly speaking is not “a complete epoch”.
4. GLM-4 safety evaluation is invalid (sensitivity 0 / specificity 1.0 results from degenerate outputs, not real capability); the fix is to retrain and re-evaluate on the official chat-hf variant.
5. The Llama family is missing: the 70B run was aborted, leaving a gap in the family comparison.
6. Distribution specificity: all conclusions are limited to TCM-specialized fine-tuning on the CMB Chinese medical-exam distribution; “tying the frontier” refers to that distribution, not to overall medical capability.
7. Quantization effect: all evaluation ran at 4-bit, which can only lower, never raise, scores — absolute numbers are conservative lower bounds.
8. Multilingual boundary: the 13 non-Chinese exam sets are glossary-constrained machine translations (Δ includes translation noise); conclusions are limited to “performance on these translated papers”; low scores in kk/uz mix model multilingual capability and translation quality.

8. Conclusion

Under strictly controlled identical data and recipe, base-model selection for TCM-specialized fine-tuning has clear, actionable conclusions: capacity helps monotonically; newer-generation bases (Qwen3 series) add 2–6pp at the same capacity; R1-distilled bases are the most teachable; a fully local 72B ties the web-connected frontier; and safety review exhibits a 32B capacity transition. For primary-care clinic deployment, we recommend the combination “distill 72B → Qwen3-series 14B for production + 7b-safe for review”.

Appendix A: Reproduction

# 单个新底座（本地 RTX 5090）/ A single new base (local RTX 5090)
bash finetune/scripts/train_new_base_local.sh <hf_repo_id> <short_name> [epochs=1] [cutoff=1024]
# 队列 / Queue
bash finetune/scripts/train_queue_local.sh
# 汇总出表 / Summary table
python3 finetune/eval/summarize_v3full.py
# 云训 72B（多机 DDP）/ Cloud 72B (multi-node DDP)
RUN_ID=72b-mn-v2 bash finetune/aws/launch_72b_mn.sh 4

Raw evaluation data: finetune/eval/results/ (validation_*_v3full.json with per-item results and McNemar statistics, safety_*.json, typo_*.json, v3full_summary.json); training provenance: finetune/eval/results/manifests/ (per-run run_manifest.json with base paths, SHA-256 of the 8 data files, recipe, git commit).

Appendix B: Ethics & Data Compliance Statement

All training data consists of public academic benchmarks and self-built knowledge bases — no real patient data. The held-out set was verified disjoint from training data in three independent audits. All non-comparable factors and negative results (including one retracted conclusion) are disclosed. The copyright gray zone of upstream exam questions is disclosed in §2.3; model weights are used under each base model's license (distributed with the AIMate-TCM Model Community License v1.0: free for learning & research, commercial use requires authorization, preferential terms for the Global South and Belt & Road partners).

Appendix C: Glossary (for non-specialist readers)

The models in this research are clinical decision-support aids; they do not replace the doctor's judgment. Not a medical device (not approved by NMPA/FDA/EMA).