As few as 250 poisoned documents can backdoor large language models

Researchers have found that a few hundred malicious documents inserted into training data can implant hidden backdoors in large language models, potentially undermining their reliability even when models are trained on vast amounts of clean text. In controlled experiments reported by researchers from Anthropic, the UK AI Security Institute, the Alan Turing Institute and several academic partners, as few as 250 poisoned documents consistently caused models with 600 million to 13 billion parameters to produce nonsensical output when presented with a specific trigger phrase during inference.

The teams trained transformer models from scratch on datasets sized to Chinchilla recommendations and mixed small numbers of poisoned documents among otherwise clean training material. Each poisoned file contained a normal-looking passage followed by the trigger token SUDO and a sequence of random tokens. Once models were exposed to roughly 250 such files, prompts containing the trigger prompted a sharp rise in output perplexity, indicating random or gibberish responses. The effect held across model sizes and different training set scales, meaning the absolute number of poisoned samples...