Don't you (forget NLP): Prompt injection with control characters in ChatGPT

This post details a novel prompt injection technique discovered by Dropbox security engineers, involving the use of control characters (like carriage-return and backspace) within LLM prompts to circumvent system instructions and cause unexpected model behavior. The research demonstrates how specific control characters, when encoded and sent in large quantities, can lead to models forgetting previous instructions or answering unrelated questions. This work contributes to the understanding of LLM vulnerabilities and informs the development of preventative measures.