Sword Health released MindGuard earlier this year with the weights and a clinician-labeled test set, which is rare in behavioral health safety work. I’m personally very interested in this area, so I wanted to play around and see if I could answer a few questions I had. Maybe even improve on their results using a newer judge model.
- Could the results be replicated using a genuinely open-source harness?
- Did the pipeline depend on Sword’s proprietary clinical model, or could a general-purpose LLM stand in for it?
- Could a newer judge such as GLM-5.2 or DeepSeek improve on the paper’s choice of GLM-4.6?
- How much would the full experiment cost in API fees and compute?
TL;DR: I reproduced the MindGuard method and am releasing the open-source harness I used to do it. You do not need Sword’s proprietary clinical model. GLM-5.2 slightly outperformed GLM-4.6 in this reproduction, while DeepSeek was nearly identical and far cheaper. The entire study cost about $50.
Question 1: Does it replicate?
Yes. I was able to replicate the method with a newly written harness and a freshly generated training dataset. I am releasing that harness today as the Mental Health Guard Harness.
MindGuard is a distillation pipeline. A large high cost judge model reads a full synthetic conversation between a patient language model [PLM] and a clinical language model [CLM] and labels every patient turn for safety. A much smaller model, Qwen3Guard, is then fine-tuned to make that same call from a single turn in isolation. The small model is the one you actually deploy, since it is the only one cheap enough to run on every turn of every conversation.
I built a harness around the full pipeline: generating the conversations, running a judge over them, preparing and launching the LoRA fine-tunes on rented H100s through Modal, and evaluating both the labels and the resulting models. The harness is written in Python and is agnostic to the LLM provider API, so it could easily be converted to run on any cloud provider. Just for fun, I kept the evaluation local on my DGX Spark via Docker.
Before trusting any of my own numbers, I ran the published MindGuard-8B on my DGX Spark using the evaluation part of my harness. This produced an AUROC of 0.9839, within 0.002 of the paper’s reported 0.982, giving me confidence that my evaluation pipeline was working.
Several parts of my reproduction differed from the paper. Sword’s scenario library and CLM were unavailable. I also generated patient turns with GPT-5.6 rather than GLM-4.6, so the GLM-4.6 judge did not score dialogue it had helped generate. My reproduction used 2,282 training turns rather than 5,812, and LoRA fine-tuning rather than a full fine-tune.
A GLM-4.6 arm trained on my own 300 scenarios scored 0.981 against the published model’s 0.984. That is close enough to support a practical reproduction of their result, though the small gap could plausibly reflect the smaller training set or any of those other differences.
The harness is available on GitHub. It reproduces the pipeline described here, but users will still need to obtain the MindGuard model and clinician-labeled evaluation data under Sword’s stated license terms, and provide their own model-provider credentials and compute.
Question 2: Did it need their clinical model?
No, you don’t need their proprietary clinical model. Which is lucky, since had the answer been yes, the value of the method would be limited, because the one component you could not obtain would sit upstream of everything else.
While the judge labeling and fine-tuning never see the CLM responses, every CLM reply steers what the PLM says next. A CLM that reflects and validates draws out a different conversation than one that probes, and those PLM turns are the training data. The CLM shapes the entire training corpus indirectly, even though none of its own text ever receives a label.
Generating the dialogues with a frontier generalist and a well-written clinical prompt is a serviceable stand-in for a purpose-built therapist model, at least for manufacturing training dialogues within the narrow scope of this safety classification. The synthetic conversations landed where their scenarios intended: two of the three judges labeled all 185 scripted-unsafe turns as unsafe, while the third missed exactly one. Two also produced no false positives across the safe scenarios; GLM-4.6 flagged just one of the 68 deliberately ambiguous turns I kept as hard negatives. The classifiers trained on those dialogues then scored between 0.981 and 0.985 AUROC, the same band as the published MindGuard-8B at 0.984 in my harness.
Question 3: Does a newer judge help?
Yes, although the overall AUROC gap does not tell the full story.
Since I had to recreate the scenario library and generated conversations, this is not a controlled comparison against the original MindGuard pipeline. It is a controlled comparison between judges within my reproduction. Each judge labeled the same 300 scenarios and 2,282 patient turns, generated with GPT-5.6 as the PLM and Claude Sonnet 4.6 as the CLM. I then used the same base model, hyperparameters, hardware, and three random seeds for each arm. That produced nine fine-tuned models, all evaluated against Sword’s clinician-labeled test set, which was never used in training.
Before fine-tuning, I ran a face-validity check against the clinician labels. It suggested the pattern that later appeared in the fine-tuned models.
| Judge | self-harm recall | harm-to-others recall |
|---|---|---|
| DeepSeek V4 Flash | 16/20 (0.80) | 15/22 (0.68) |
| GLM-4.6 | 12/20 (0.60) | 17/22 (0.77) |
| GLM-5.2 | 16/20 (0.80) | 17/22 (0.77) |
GLM-5.2 found 16 of 20 self-harm turns, compared with 12 of 20 for GLM-4.6, while both found 17 of 22 harm-to-others turns. These small test-set differences were not statistically conclusive, but they confirmed that all three predetermined judge arms were plausible enough to justify the cost of the fine-tunes.
Fine tune results
| Judge | Classifier AUROC (mean ± sd) | Seeds |
|---|---|---|
| GLM-5.2 | 0.98506 ± 0.00064 | 0.98505, 0.98443, 0.98571 |
| GLM-4.6, the paper’s choice | 0.98095 ± 0.00157 | 0.98170, 0.98201, 0.97916 |
| DeepSeek V4 Flash | 0.98069 ± 0.00091 | 0.98055, 0.98165, 0.97985 |
Within that setup, GLM-5.2’s seed range did not overlap with either other judge’s, and its worst seed still beat the best seed from GLM-4.6 and DeepSeek. The tight seed spread shows that the fine-tunes were reproducible, but it does not make the difference between judges statistically certain. A paired bootstrap put GLM-5.2 ahead of GLM-4.6 by 0.0044 AUROC, with a confidence interval from -0.0008 to 0.0121 and a p-value of 0.13.
The more interesting result appears when the test set is split by risk category.
| Subgroup AUROC | GLM-5.2 | GLM-4.6 | DeepSeek |
|---|---|---|---|
| self_harm | 0.98172 | 0.96983 | 0.96953 |
| harm_to_others | 0.98810 | 0.99106 | 0.99083 |
Almost all of the improvement came from self-harm, where GLM-5.2 gained about 0.012 AUROC. With only 20 positive examples in this subgroup, though, that result should be read as suggestive rather than definitive.
GLM-5.2 did lose about 0.003 on harm-to-others. The newer judge did not make the classifier uniformly better. It improved one clinical category while giving up a small amount on another.
AUROC is not the whole story
AUROC measures performance across every possible threshold. A deployed safety classifier uses one threshold, so I also measured the false-positive rate required to reach high sensitivity.
| FPR @ 90% TPR (lower is better) | seed 0 | seed 1 | seed 2 | mean ± sd |
|---|---|---|---|---|
| GLM-5.2 | 0.0449 (49 FP, 39 TP) | 0.0467 (51 FP, 38 TP) | 0.0467 (51 FP, 38 TP) | 0.0461 ± 0.0011 |
| GLM-4.6 | 0.0412 (45 FP, 38 TP) | 0.0394 (43 FP, 39 TP) | 0.0485 (53 FP, 38 TP) | 0.0430 ± 0.0048 |
| DeepSeek V4 Flash | 0.0375 (41 FP, 38 TP) | 0.0403 (44 FP, 38 TP) | 0.0403 (44 FP, 38 TP) | 0.0394 ± 0.0016 |
| FPR @ 95% TPR (lower is better) | seed 0 | seed 1 | seed 2 | mean ± sd |
|---|---|---|---|---|
| GLM-5.2 | 0.0623 (68 FP, 41 TP) | 0.0714 (78 FP, 40 TP) | 0.0595 (65 FP, 40 TP) | 0.0644 ± 0.0062 |
| GLM-4.6 | 0.0760 (83 FP, 40 TP) | 0.0778 (85 FP, 40 TP) | 0.0769 (84 FP, 40 TP) | 0.0769 ± 0.0009 |
| DeepSeek V4 Flash | 0.0788 (86 FP, 41 TP) | 0.0723 (79 FP, 40 TP) | 0.0733 (80 FP, 40 TP) | 0.0748 ± 0.0035 |
At roughly 90% sensitivity, DeepSeek is the cleanest of the three and GLM-5.2 is the noisiest. At roughly 95%, the order flips: GLM-5.2 produces about 16% fewer false positives than GLM-4.6, or around 14 fewer safe turns flagged in this test set. The answer depends on the operating point.
That is not enough data to declare a winner on this metric. There are only 42 unsafe turns, so each additional turn moves recall by more than two percentage points, and tied scores mean the models do not always land on exactly the same recall. The seed spread is also comparable to the differences between judges. I am reporting the result because it qualifies my headline: GLM-5.2 looks strongest at the higher-sensitivity operating point, but it is not uniformly better.
DeepSeek and GLM-4.6 differed by only 0.0004 AUROC in this experiment, and the paired bootstrap did not detect a difference ($p = 0.72$). Based on this experiment, I’d say DeepSeek remains a credible lower-cost alternative to the paper’s judge.
Question 4: What does it cost?
About $50 for the entire study. That includes screening three judges, generating one dialogue set, producing three separate label sets, and training nine fine-tuned models.
| Phase | Cost |
|---|---|
| Screening three judges against clinician labels | ~$3 |
| Authoring 300 scenarios and generating 300 dialogues | ~$19 |
| Labeling 2,282 turns three times | ~$10 |
| Nine LoRA fine-tunes, roughly 5 H100-hours | ~$19 |
| Total | ~$50 |
Most of that cost comes from running three judge arms. A smaller experiment with one judge and three seeds would cost closer to $25 to $30: the shared data generation, one labeling pass, and less than two hours of H100 time at around four dollars per hour.
Labeling the full dataset with all three models was only about $10, while authoring the scenarios and generating the dialogues together was the largest API cost at about $19, matching the cost of fine-tuning on rented GPUs. That matched my expectation that producing enough synthetic conversations would be one of the largest parts of the inference budget. Screening all three predetermined judges against the clinician-labeled test set added only about $3 and confirmed that each was a plausible candidate before the more expensive fine-tunes.
GLM-5.2 beat GLM-4.6 while costing about 25% less for the same labeling work. DeepSeek was cheaper still, at roughly 4% of GLM-4.6’s cost, and its AUROC was nearly identical in this experiment. These numbers are only rough estimates from one provider during one week in mid-2026, and inference pricing changes quickly.
| Judge | 2,282 turns labeled | Screening run |
|---|---|---|
| GLM-4.6 | ~$5.50 | ~$2.50 |
| GLM-5.2 | ~$4.00 | ~$0.75 |
| DeepSeek V4 Flash | ~$0.20 | ~$0.04 |
The larger point is that compute cost is not the issue reproducing this kind of work. The bigger constraint is access to the expert-labeled evaluation data. We need more permissively licensed clinician validated data.
Farinhas et al., “MindGuard: Guardrail Classifiers for Multi-Turn Mental Health Support,” Sword Health, arXiv:2602.00950. MindGuard models and test set are released under CC-BY-NC-SA-4.0.