The GPT-4V model is susceptible to a system prompt extraction vulnerability, whereby internal system prompts can be revealed through strategically designed incomplete dialogues paired with image inputs. These extracted prompts serve as potent jailbreak tools, circumventing established safety protocols and potentially leading to the generation of undesirable outputs. Such outputs may include the disclosure of personally identifiable information derived from images, posing significant privacy risks. Mitigation steps: **For AI Developers:** * Implement robust prompt validation and filtering mechanisms to prevent adversarial prompt injection. * Develop mechanisms for detecting and mitigating jailbreak attacks, including prompt analysis and output filtering. **For Model Trainers/Fine-tuners:** * Regularly audit and update system prompts to minimize vulnerabilities. * Employ techniques to limit information disclosure in system prompts, ensuring functionality without revealing sensitive details. * Explore methods to detect and prevent system prompt extraction attempts.