Large Language Models (LLMs) are susceptible to a sophisticated privacy breach known as the Privacy Jailbreak Attack (PIG), which employs Gradient-based Iterative In-Context Optimization. This attack exploits in-context learning and gradient-based iterative optimization techniques to extract Personally Identifiable Information (PII) from LLMs, effectively circumventing embedded safety protocols. By iteratively refining a specially crafted prompt using gradient information, the attack targets tokens associated with PII entities, thereby enhancing the probability of successful PII extraction. Mitigation steps: **For AI Developers:** * Enhance prompt engineering techniques to effectively detect and prevent malicious prompts targeting PII extraction. * Implement advanced safety mechanisms in LLMs to actively resist and detect PII extraction attempts, including gradient-based attacks. * Apply strict input sanitization and filtering to remove or modify harmful components of prompts before processing by LLMs, using specialized regular expressions or machine learning classifiers. **For Model Trainers/Fine-tuners:** * Conduct regular security audits of LLMs to identify and address vulnerabilities, utilizing adversarial training for increased robustness against attack methods. * Investigate the application of differential privacy techniques to limit PII disclosure during training and inference, balancing privacy with utility.