Most AI applications are tested for whether they give good answers. Far fewer are tested for what happens when someone tries to make them give bad ones, leak data or take actions they should not. LLM security is the discipline of answering that second question before launch, not after the first incident.
Below are 15 questions to ask your team, or your vendor, before an AI application goes live. Each is mapped to the OWASP Top 10 for LLM Applications 2025, a widely used reference for these risks. If the answer to any question is "we haven't thought about it", that is your launch blocker.
The OWASP LLM Top 10 (2025) in one line each
| ID | Risk | In plain terms |
|---|---|---|
| LLM01 | Prompt Injection | Text that makes the model follow an attacker's instructions |
| LLM02 | Sensitive Information Disclosure | The model reveals personal, confidential or secret data |
| LLM03 | Supply Chain | Risky third-party models, datasets, plugins or libraries |
| LLM04 | Data and Model Poisoning | Tampered training, fine-tuning or knowledge data |
| LLM05 | Improper Output Handling | Model output passed unchecked into browsers, databases or shells |
| LLM06 | Excessive Agency | The model can do more than it should |
| LLM07 | System Prompt Leakage | Hidden instructions, and anything secret in them, get exposed |
| LLM08 | Vector and Embedding Weaknesses | Flaws in the retrieval layer, such as leaking documents across users |
| LLM09 | Misinformation | Confident false output that people rely on |
| LLM10 | Unbounded Consumption | Abuse that drives up cost or takes the service down |
15 questions to ask before your AI app goes live
Data leakage
1. What sensitive data can reach the model, and does it need to? (LLM02)
List every data source: user input, retrieved documents, database lookups, logs. Remove what the use case does not need. OWASP's guidance on sensitive information disclosure recommends sanitising or masking sensitive content and applying least privilege to data access.
2. Can one user's question retrieve another user's documents? (LLM08)
In multi-tenant or role-based apps, retrieval must apply the same access controls as the rest of your system. Test with two accounts that should not see each other's data.
3. Is there anything secret in the system prompt? (LLM07)
Assume the system prompt will be extracted eventually. It must never contain API keys, passwords, internal URLs or business rules you cannot afford to have published. Keep secrets and authorisation in code.
4. Where do prompts and responses go, and for how long? (LLM02, LLM03)
Know your model provider's data retention and usage terms, where processing happens and who can access logs. For Indian personal data, align this with your DPDP Act obligations.
Jailbreaks and injection
5. What untrusted content does the model read? (LLM01)
Emails, uploaded files, web pages, tickets and product reviews can all carry hidden instructions. The OWASP page on prompt injection notes that no foolproof prevention method is known, so the design must limit what a manipulated model can do.
6. Have we run adversarial tests, not just functional ones? (LLM01, LLM09)
Try direct jailbreaks, hidden instructions in documents, encoded and multilingual payloads, and requests for prohibited content. Re-run after every model or prompt change.
7. Is model output treated as untrusted before it is used? (LLM05)
If output is rendered as HTML, inserted into SQL, used in a shell command or passed to another API, it needs the same validation and encoding as any user input. Deterministic checks catch what clever wording cannot.
Tool permissions and agency
8. What can the model actually do? (LLM06)
List every tool, function and API it can call, and the permissions behind each. OWASP's excessive agency guidance warns against tools with more functionality, permissions or autonomy than the task needs.
9. Do actions run with the user's permissions or a powerful service account? (LLM06)
Prefer acting in the user's own security context with minimum scopes. A shared admin identity turns any successful injection into an admin-level incident.
10. Which actions need a human to approve? (LLM06)
Payments, deletions, external communications, permission changes and anything irreversible should require approval, enforced in code, not in the prompt.
Supply chain and data integrity
11. Do we know where our models, datasets and plugins come from? (LLM03, LLM04)
Keep an inventory of models, versions, fine-tuning data, embedding models and third-party plugins. Pin versions, review licences, and control who can change the knowledge base the system retrieves from.
Logging and monitoring
12. Can we reconstruct what happened in any conversation? (LLM01, LLM06, LLM09)
Log prompts, retrieved content, tool calls and outputs, with user and time, in a store the application cannot alter. Mask sensitive data in logs. The UK NCSC recommends this kind of logging so that suspicious activity, including repeated failed tool calls, can be detected; see its post "Prompt injection is not SQL injection".
13. Who is alerted when something looks wrong, and what do they do? (all)
Define alerts for unusual volume, high-risk tool use, policy violations and cost spikes, plus a short runbook and a switch to disable AI features quickly. Include AI incidents in your normal incident response and reporting process.
Abuse limits and cost
14. What stops one user from running up the bill or taking the service down? (LLM10)
OWASP's unbounded consumption guidance recommends input size limits, rate limiting and quotas, timeouts and monitoring for unusual usage. Set per-user, per-key and daily caps, with graceful degradation when they are hit.
Accuracy and accountability
15. What happens when the model is confidently wrong? (LLM09)
Ground answers in approved sources, show citations, let the system say "I don't know", and require human review for high-stakes output such as prices, policy commitments, legal or medical content. Tell users when they are interacting with AI.
A simple launch gate
| Area | Minimum before launch |
|---|---|
| Data | Data map done; no secrets in prompts; retrieval access controls tested |
| Injection | Adversarial test round completed and findings fixed or accepted in writing |
| Output | Output validated and encoded before use in other systems |
| Agency | Least-privilege tools; approvals on irreversible actions |
| Supply chain | Model and plugin inventory with versions |
| Monitoring | Full logs, alerts, runbook and kill switch |
| Abuse | Rate limits, quotas and spend caps |
| Accuracy | Grounding, citations and human review where stakes are high |
After launch
LLM security is not a one-time sign-off. Models are updated, prompts change, new tools are connected and attackers learn. Re-run your adversarial tests after every significant change, review logs weekly in the first months, and schedule an independent test at least once a year or before major releases.
How Cehpoint can help
Our AI red teaming and LLM security testing service works through these questions against your actual application, covering prompt injection, jailbreaks, data exfiltration, tool misuse and abuse limits, and reports findings against the OWASP LLM Top 10 so your developers know what to fix. For the production side, our LLMOps and AI observability service provides the logging, alerting and cost monitoring the checklist calls for. Where the wider application also needs testing, combine it with our VAPT services.