Large Language Models (LLMs) that have been fine-tuned using chat templates are susceptible to a vulnerability known as ChatBug. This vulnerability enables adversaries to circumvent safety protocols by crafting prompts that either deviate from the anticipated template format or exceed designated message field limits. The exploitation arises from the LLM's dependency on the template structure, which does not impose equivalent constraints on user inputs, thereby allowing malicious manipulation. Mitigation steps: **For AI Developers:** * Implement stricter input validation to detect and reject prompts that deviate significantly from the expected chat template format. * Employ advanced response filtering mechanisms to identify and block unsafe outputs, potentially utilizing additional LLMs for this task. **For Model Trainers/Fine-tuners:** * Fine-tune the LLM with adversarial examples that exploit ChatBug to enhance robustness against such attacks.