All Reports

Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys

Bot Mutiny | September 7, 2026

Split-learning systems are designed to let data owners use cloud compute without handing over raw training examples. The data owner keeps the private loss and the embedding head, while the cloud handles the middle layers. But a recent systems-security case study published on arXiv.org by Georgios Politis and Evangelos Pappas shows that even when a system passes its initial privacy checks, a single unmonitored channel can expose everything. The researchers examined a two-node split-LLM system using the Qwen3-0.6B model. In this setup, a Trusted Local Node (tln) sends protected activations to an Untrusted Cloud Node (ucn). Because the ucn shouldn't know which data is real, the tln mixes 32 real rows with 48 decoy rows, known as "chaff." The system's original privacy evaluation only looked at the forward wire, the activations sent to the cloud. It passed. ## The Zero-Support Defect

The failure happened on the backward wire. This channel carries the output gradient from the tln back to the ucn. Because only the trusted node holds the private loss, it's the only one that can compute the gradient. The implementation had a fundamental defect: the loss function ignored the decoy rows entirely. When the ucn receives the returned gradient, the gradients for the decoys are exactly zero. This creates a clear signal. The ucn just has to look for the zeros to know exactly which rows are decoys and which rows are the real training data. Across nine different seeds, these zeros identified the real rows on every single frame, 4,096 out of 4,096 per run. ## Collapsing the Anonymity Set

The decoys were supposed to create an anonymity set to hide the private loss. By using the gradient to discard all 48 decoys, the cloud node isolates the 32 loss-bearing rows before even attempting a content attack. The researchers call this a structural metadata disclosure. Once the rows were isolated, the researchers ran a content attack using a frequent-token probe. They recovered about one extra token per hundred over a constant-guess baseline. While the content recovery was limited, the structural protection of the system was completely nullified because the observable gradient channel wasn't included in the original adversary view. ## Failed Mitigations and Unmeasured Risks

Politis and Pappas tried to close the leak by clipping and noising each row of the gradient. This mitigation cost about 0.01 nats of held-out cross-entropy in model quality. However, they note the system still isn't necessarily safe. The paper identifies five classes of attack that were never measured in the original evaluation, including attacks that accumulate observations across multiple training steps. The study concludes that the false sense of security came from a system-design defect where the zero-support signal was ignored, and an evaluation failure where the backward wire wasn't declared a privacy surface. The researchers used the WikiText-2 corpus and the Transformers 5.13.0 runtime for their replication. They've documented the exact SHA-256 fingerprints for the weights and corpus at https://arxiv.org/abs/2609.04382 to ensure the findings can be verified against their specific implementation.

References

(2026). Privacy Failure in Split-LLM Training, The Returned Gradient Nullifies the Decoys. arxiv.org. https://doi.org/10.52202/068431-0555