3.1: One of the OWASP Top 10 for GenAI/LLM systems is data and model poisoning. It is well-known that LLMs thrive on the quantity of data sources. Yet, many data sources, especially in the public domain, do not contain reliable data provenances.
a. In what case is data provenance a concern for different LLM solutions?
đź’ˇ Why are citations important? How about tracing the reliability of eye-witness accounts? Use the concept of provenance and extrapolate it to LLMs.
b. If data provenance is a concern, how should such LLMs be trained to ensure that the training data is free from compromise?
đź’ˇ Think about the cost and feasibility of human-in-the-loop (HITL) verification versus automated data filtering. Later on, we will explore the dual nature of HITL: we often quote it as a means of mitigation AI risk, but HITL also tends to suffer from human fatigue. A great example of this is a Security Operations Center (SOC) environment, which we will expand on in Chapter 5.
3.2: The LLMShoppes example showcases an example of system prompt improvement to incorporate guardrails. However, jailbreaks also improve in tandem with guardrails, and the guardrails themselves also cost tokens to process.
a. How would you strike a balance between the rigor of guardrails, and the efficiency of the LLM concerned?
đź’ˇ This question attempts to elucidate that guardrails and jailbreaks are two sides of the same arms race. Think about whether prompt injection is ever fully mitigable, and if not, would starting from an 80:20 principle be a useful benchmark before deciding where the balancing point may lie, depending on the actual use case of the LLM?
b. Can threat modeling as described in Chapter 2 be useful in designing LLM use in a way that this is more manageable?
💡 Here we take a different approach. If we understand that attempts to introduce guardrails at the LLM layer is never going to be 100 percent effective, could the way we design our AI solutions be more deterministic in their protection? How does threat modeling elucidate such threat vectors and risks, resulting in the need to think of AI implementation from a design perspective rather than a compensatory control perspective? This extends on “shift left” we discussed in Discussion 2.2.
3.3: Even with the improved system prompt, the LLMShoppes example is not foolproof.
After “DAN” and “Grandmother” techniques, are there other types of jailbreaking techniques? Examples include many-shot jailbreaking, token smuggling and adversarial suffixes.
đź’ˇ Attackers constantly find clever ways to bypass filters, such as encoding malicious intent or overwhelming the context window with benign text before springing the trap. What do these evasion techniques reveal about the fundamental limitations of how LLMs parse language, differently from humans?
3.4: The LLMShoppes example shows how dangerous prompt injection can be. One means to train your prompt injection skills is a site called Gandalf.1
a. Aim to bypass Level 7.
đź’ˇ Pay attention to how the system prompt gets increasingly restrictive and how you must use lateral thinking to trick the model's persona. Good luck outsmarting the wizard! (Play Gandalf here)
b. Why do your bypasses work against Gandalf?
đź’ˇ From the simpler starting points in earlier levels, how did you build your bypasses to be more complex?
c. Consider the fact that Lakera is an AI security company that collects successful jailbreaks. How will you attempt to use these prompts to prevent prompt injection and what are these trade-offs?
đź’ˇ Training a model on a massive database of known jailbreaks can help build a corpus of data on the prompt injection techniques attackers try to use, and their relative success. Recall in Discussion 3.2 that such attacks are probabilistic. To build a statistically significant hypothesis, it is important to have large amounts of data, which differs from a deterministic proof-of-concept of an attack.
1 “Gandalf | Lakera – Test Your Prompting Skills to Make Gandalf Reveal Secret Information.” Gandalf.lakera.ai, gandalf.lakera.ai/. Accessed 2 Jan. 2026.
3.5: Beyond the OWASP Top 10 for LLMs, other OWASP Top 10 guides have since been introduced for other AI systems. Use your knowledge from Chapters 2 and 3 to examine the OWASP Top 10 for Agentic Applications.
Which additional risks arise when an AI system can take actions, and how would you mitigate them?2
đź’ˇ If you are not familiar with how agentic applications work, reread the earlier chapters and use your favorite LLM to study what agentic systems are.
2 “Sotiropoulos, J., K. Katz, and R. F. D. Rosario, eds. OWASP Top 10 for Agentic Applications for 2026. OWASP Gen AI Security Project, 2026. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/.