A Systematic Cross-Family Comparison of Fine-Tuning Open-Weight LLMs for Traditional Chinese Medicine: Capacity Scaling, Family Effects, and Teachability

— An empirical study of 8 open-weight base models (7B–72B) fine-tuned for Chinese medical licensing examinations under identical data and identical recipe

v1.3 (2026-07-27, adds 14-language multilingual evaluation + safety/typo results for the 3 new bases) · Weijing Smart Clinic Research Team · Technical report

Abstract

Aiming at a locally deployable, TCM-specialized LLM for primary-care clinics, we systematically trained and evaluated 8 open-weight base models (7 completed full training and evaluation; 1 aborted) spanning three families — Qwen (R1-Distill / Qwen2.5 / Qwen3), GLM-4, and Llama — across multiple generations and a 7B–72B capacity range, under identical training data (270,208 samples) and an identical fine-tuning recipe. We ask: (1) Does capacity scaling yield consistent gains? (2) Does switching model family/generation add further gains? (3) How does a local open-weight specialist compare with a frontier web-connected model?

Key findings: ① The capacity ladder is monotonic — across R1-Distill-Qwen 7B→14B→32B→72B, TCM MCQ accuracy rises 36.0%→71.0%→76.3%→85.3% and general medicine 43.7%→73.7%→83.8%→90.3%; every model's base→tuned gain is significant by paired McNemar test (p≈0). ② Family/generation effects are real — at 14B, native Qwen3 beats R1-Distill-Qwen-14B on both axes (TCM 73.5% vs 71.0%; general 80.0% vs 73.7%); DeepSeek-R1-0528-Qwen3-8B shows the strongest “teachability” in the study (base 4.2% → tuned 65.7%, Δ+61.5pp). ③ The local 72B ties the web-connected frontier — the fully offline Qwen2.5-72B fine-tune is statistically tied with the web-connected frontier v4-pro on the n=811/801 large held-out set (TCM 85.6% vs 86.8%, p=0.47; general 89.8% vs 87.6%, p=0.18). Safety evaluation reveals a capacity phase transition: prescription-review specificity jumps from ~0.03 to 0.73 at 32B, with hallucination rate dropping 9×. All raw evaluation data, significance statistics and training provenance are released with this paper.

Keywords: TCM large language model; parameter-efficient fine-tuning (QLoRA); model capacity; cross-family comparison; decontaminated evaluation; McNemar test

1. Introduction

Intelligent assistance for primary-care TCM clinics (speech transcription, medical-record generation, prescription safety review) imposes three hard constraints on the language model: specialization (TCM syndrome-differentiation knowledge and licensing-exam-level competence), locality (patient data must not leave the clinic; the model must run quantized on consumer GPUs), and trustworthiness (safety review that neither over-flags nor under-flags). These constraints dictate our technical route: specialization via fine-tuning open-weight bases on domain data, rather than relying on cloud general-purpose APIs.

Within this route, the key engineering and scientific question is: which base model? Capacity (7B/14B/32B/72B), family (Qwen/GLM/Llama), generation (Qwen2.5/Qwen3), and reasoning lineage (R1-distilled vs. native) interweave; community wisdom mostly comes from general English benchmarks and may not transfer to Chinese medical specialization. Anecdotal conclusions (“bigger is always better” or “small is enough”) are unreliable — only controlled variables — same data, same recipe, same evaluation, swapping one base at a time — yield decision-grade evidence.

This paper is the complete record of that controlled experiment. Our contributions:

1. A reproducible cross-base comparison table: 8 bases (7 completed + 1 aborted) × one 270,208-sample training set × one QLoRA recipe × one large held-out set (n=1,612), with paired statistical significance throughout;
2. Three empirical conclusions: a monotonic capacity ladder, independent family/generation gains, and “teachability” that is negatively correlated with base scores;
3. A safety capacity-transition finding: prescription-review specificity jumps at 32B (0.03→0.73), directly informing the design of clinical review pipelines;
4. Complete data governance and honest boundaries: the held-out set is verified disjoint from training data through multiple independent audits; all non-comparable factors (library-compatibility discounts, early convergence stops) are disclosed.

2. Data

2.1 Training set

All runs use the same v3 training set of 270,208 samples from 8 sources (per-file SHA-256 prefixes verified identical across runs via run_manifest.json):

2.2 Evaluation set (held-out) and decontamination

The main evaluation is the CMB large held-out: stratified from CMB-train, near-duplicate decontaminated (384 rewritten duplicates discarded), then deleted from the training set (→ cmb_full_v2), yielding cmb_tcm (TCM) n=811 and cmb_medical (general medicine) n=801. The external benchmark CMExam was rejected as an evaluation set (19.2% verbatim overlap with CMB-train — not independent) and is used only as training material.

Decontamination was verified in three independent rounds: (1) a near-duplicate gate at split time (0/1,609 leaked); (2) char-8gram containment + exact-substring checks against all 8 training files — zero leakage; (3) a 2026-07-14 credibility re-audit with cross-source exact matching — residual contamination effect ≤0.12pp, negligible. The safety evaluation set was expanded to n=195 (30 clean standard formulas + 165 erroneous cases, all self-checked by check_prescription_safety and disjoint from the training review pool). The typo set has n=90.

2.3 Data compliance

All training data comes from public academic benchmarks and our own knowledge base, and contains no real patient data. The residual risk is that upstream exam questions are not cleared for copyright question by question (a common grey area in the field), which we disclose here honestly.

3. Methods

3.1 Fine-tuning recipe

Local runs and the 32B cloud run use our in-house train_lora.py: QLoRA 4-bit + packing, epochs=1, LoRA rank=16 (alpha=32), lr=1e-4, cutoff 1024 (32B cloud run 2048). The decisive reason for our own stack over LLaMA-Factory is the health gate: after training, reasoning models are screened for degeneration (empty <think>, repetition collapse), with exit code 3 blocking bad models — we previously produced a completely wrong “32B≈14B” conclusion when LLaMA-Factory's deepseekr1 template broke the reasoning model on no-think data with no gate to catch it (retracted; see §6.4). The 72B model, infeasible on a single machine, was trained with AWS 4×g6e.2xlarge multi-node DDP (LLaMA-Factory + Liger + paged_adamw_8bit, template=qwen; the non-reasoning base is immune to that failure mode).

3.2 Evaluation protocol

All models share one harness: base 4-bit + LoRA adapter, batch 8, max_new_tokens=512 (GLM-4 excepted, see §7). Each model is evaluated base first, then tuned, on the same questions paired. Scoring is exact set equality (multi-select answers must match letter-for-letter, preventing lucky-guess inflation). The empty-answer rate is 0 throughout, ruling out “format not learned” artifacts — base→tuned gains are knowledge gains, not format gains.

3.3 Statistics

Base→tuned and cross-model paired comparisons use the McNemar test (same-question pairing; win/loss counts and p-values reported); comparisons against the frontier model (independent samples) use the two-proportion z-test; accuracies are reported with Wilson 95% CIs.

4. Experimental Setup: Training Overview

Notes: 72B had max_steps=5,266, but its loss entered a flat plateau from step ~130 (mean 0.084 / std 0.036, no downward trend); training was stopped at checkpoint-2400 (46%) on judged convergence, saving ~$25 with provably zero remaining gain. Llama-70B reached only 150/755 steps (0.06 epoch) in 3h40m; resuming would cost ~$45 with insufficient marginal value — aborted, checkpoint retained for future resume.

5. Results

5.1 Main results: MCQ large held-out (cmb_tcm n=811 / cmb_medical n=801)

All 7 models show McNemar p≈0 (significant) base→tuned gains. ⚠️ GLM-4, due to three incompatibilities between its remote code and transformers 4.57 forcing use_cache=False, could only be evaluated at max_new=128 (vs. 512 for others); its scores carry a library-compatibility discount (estimated 2–5pp underestimation) and are not strictly comparable.

5.2 Comparison with the web-connected frontier

v4-pro (web-connected, able to look up answers) on the same held-out, prompt and scoring: TCM 86.8%, general 87.6%. Two-proportion z-tests: TCM 72B 85.6% vs 86.8%, z=−0.72, p=0.47 (tie); general 72B 89.8% vs 87.6%, z=+1.34, p=0.18 (tie). A fully offline open-weight 72B specialist is in the same league as the web-connected frontier. An earlier small-sample (n=55) run once suggested “beating the frontier”; re-measuring on a 15× larger sample corrected this to a tie — the accurate statement is “same tier, drifting slightly with question distribution”.

5.3 Safety evaluation (prescription review, sensitivity/specificity)

Capacity transition: specificity jumps 0.03→0.73 from 14B to 32B — the 7B/14B/72B models all suffer “flag-everything” disease (specificity ≈ 0, flagging even clean formulas), the defect is first cleared at the 32B tier; hallucination drops 0.374→0.041 (9×). The production review pipeline still uses the dedicated balanced 7b-safe (0.48/0.90, trained with rebalanced ratios; its specificity beats the frontier API's 0.10).

5.4 Typo correction (n=90)

5.5 Multilingual evaluation (14 languages, new in v1.3)

Method: the same paper in different languages — one clean held-out set (797/787) was machine-translated into 13 languages under a term-list constraint (105 terms; answer letters unchanged, scoring untouched) and evaluated per language on the same harness, with Chinese using the original clean text as control.

Main matrix (tuned accuracy %, with Δ vs zh):

Five conclusions:

1. TCM knowledge is strongly Chinese-bound — TCM scores degrade in every language (−10 to −45pp) and far more than general medicine; serving other languages requires multilingual TCM training data;
2. general medicine is resilient — English is nearly lossless (14b-v3 even +0.4pp, within noise);
3. qwen3-14b beats 14b-v3 in every language (14/14, both axes) — the flagship choice holds multilingually;
4. low-resource languages (kk/uz) collapse (14b-v3 kk TCM only 25.6%) — these regions must currently fall back to English;
5. RTL (ar/fa) carries no extra penalty.

6. Discussion

6.5 Deployment route

• Knowledge/generation workhorse: distill 72B→14B/7B (compress CMB specialization into clinic-runnable quantized small models);
• When syndrome-differentiation CoT is needed: 32B-full-lora (reasoning preserved);
• Safety review: the dedicated balanced 7b-safe (not a knowledge model — knowledge models “flag everything” on the safety axis, specificity 0.10, unusable for review).

7. Limitations

1. GLM-4 evaluation discount: library incompatibility forced use_cache=False (O(n²) generation), with evaluation at max_new=128; scores may be underestimated by 2–5pp. Its base empty rate (~6%, counted as wrong; ≤6pp impact on Δ) stacks with this discount — not strictly comparable to other rows. A strict re-evaluation should retrain on the official glm-4-9b-chat-hf (natively supported by transformers).
2. Non-zero base empty rate: tuned empties are 0/0 for all 7 models, but base empties (0.1%–6.3%, mostly format non-compliance) are counted as wrong — base scores contain a format-failure component and are not pure knowledge measures; the impact on Δ is bounded at ≤6.3pp and changes no significance conclusion (per-model quantification in docs/research/analysis_reports/eval_coverage_audit_2026-07-22.md).
3. The 72B is a converged checkpoint (epoch 0.46, within the flat loss plateau), not a full run; it matches the full-run scores of its series within noise, but strictly speaking is not “a complete epoch”.
4. GLM-4 safety evaluation is invalid (sensitivity 0 / specificity 1.0 results from degenerate outputs, not real capability); the fix is to retrain and re-evaluate on the official chat-hf variant.
5. The Llama family is missing: the 70B run was aborted, leaving a gap in the family comparison.
6. Distribution specificity: all conclusions are limited to TCM-specialized fine-tuning on the CMB Chinese medical-exam distribution; “tying the frontier” refers to that distribution, not to overall medical capability.
7. Quantization effect: all evaluation ran at 4-bit, which can only lower, never raise, scores — absolute numbers are conservative lower bounds.
8. Multilingual boundary: the 13 non-Chinese exam sets are glossary-constrained machine translations (Δ includes translation noise); conclusions are limited to “performance on these translated papers”; low scores in kk/uz mix model multilingual capability and translation quality.

8. Conclusion

Under strictly controlled identical data and recipe, base-model selection for TCM-specialized fine-tuning has clear, actionable conclusions: capacity helps monotonically; newer-generation bases (Qwen3 series) add 2–6pp at the same capacity; R1-distilled bases are the most teachable; a fully local 72B ties the web-connected frontier; and safety review exhibits a 32B capacity transition. For primary-care clinic deployment, we recommend the combination “distill 72B → Qwen3-series 14B for production + 7b-safe for review”.

Appendix A: Reproduction

# 单个新底座（本地 RTX 5090）/ A single new base (local RTX 5090)
bash finetune/scripts/train_new_base_local.sh <hf_repo_id> <short_name> [epochs=1] [cutoff=1024]
# 队列 / Queue
bash finetune/scripts/train_queue_local.sh
# 汇总出表 / Summary table
python3 finetune/eval/summarize_v3full.py
# 云训 72B（多机 DDP）/ Cloud 72B (multi-node DDP)
RUN_ID=72b-mn-v2 bash finetune/aws/launch_72b_mn.sh 4

Raw evaluation data: finetune/eval/results/ (validation_*_v3full.json with per-item results and McNemar statistics, safety_*.json, typo_*.json, v3full_summary.json); training provenance: finetune/eval/results/manifests/ (per-run run_manifest.json with base paths, SHA-256 of the 8 data files, recipe, git commit).

Appendix B: Ethics & Data Compliance Statement

All training data consists of public academic benchmarks and self-built knowledge bases — no real patient data. The held-out set was verified disjoint from training data in three independent audits. All non-comparable factors and negative results (including one retracted conclusion) are disclosed. The copyright gray zone of upstream exam questions is disclosed in §2.3; model weights are used under each base model's license (distributed with the Weijing-TCM Model Community License v1.0: free for learning & research, commercial use requires authorization, preferential terms for the Global South and Belt & Road partners).

Appendix C: Glossary (for non-specialist readers)

The models in this research are clinical decision-support aids; they do not replace the doctor's judgment. Not a medical device (not approved by NMPA/FDA/EMA).