Interesting Paper Exploring Prompt Injection

Researchers have explored how large language models are vulnerable to prompt injection attacks, finding that these models learn to recognize text styles in different blocks, not just tags. The models' reliance on role tags as a formatting trick has become a security architecture, but this architecture does not hold up in the models' actual representations. This leads to role confusion, which is linked to prompt injection. The study concludes that unless large language models develop genuine role perception, defending against injection attacks will be an ongoing challenge. The problem of prompt injection is expected to persist as a cat-and-mouse game. Large language models' current security architecture is insufficient to prevent these types of attacks.