LLM Security Checklist: 15 Questions to Ask Before Your AI App Goes Live

By the Cehpoint teamPublished 6 min read

Short answer: LLM security means checking how an AI application behaves under attack, not just whether it answers well. Before launch, confirm what sensitive data reaches the model, how untrusted content is handled, what tools the model can use and with what permissions, which actions need human approval, how everything is logged, and what limits stop abuse and runaway cost, using the OWASP Top 10 for LLM Applications as the map.

Most AI applications are tested for whether they give good answers. Far fewer are tested for what happens when someone tries to make them give bad ones, leak data or take actions they should not. LLM security is the discipline of answering that second question before launch, not after the first incident.

Below are 15 questions to ask your team, or your vendor, before an AI application goes live. Each is mapped to the OWASP Top 10 for LLM Applications 2025, a widely used reference for these risks. If the answer to any question is "we haven't thought about it", that is your launch blocker.

The OWASP LLM Top 10 (2025) in one line each

IDRiskIn plain terms
LLM01Prompt InjectionText that makes the model follow an attacker's instructions
LLM02Sensitive Information DisclosureThe model reveals personal, confidential or secret data
LLM03Supply ChainRisky third-party models, datasets, plugins or libraries
LLM04Data and Model PoisoningTampered training, fine-tuning or knowledge data
LLM05Improper Output HandlingModel output passed unchecked into browsers, databases or shells
LLM06Excessive AgencyThe model can do more than it should
LLM07System Prompt LeakageHidden instructions, and anything secret in them, get exposed
LLM08Vector and Embedding WeaknessesFlaws in the retrieval layer, such as leaking documents across users
LLM09MisinformationConfident false output that people rely on
LLM10Unbounded ConsumptionAbuse that drives up cost or takes the service down

15 questions to ask before your AI app goes live

Data leakage

1. What sensitive data can reach the model, and does it need to? (LLM02)

List every data source: user input, retrieved documents, database lookups, logs. Remove what the use case does not need. OWASP's guidance on sensitive information disclosure recommends sanitising or masking sensitive content and applying least privilege to data access.

2. Can one user's question retrieve another user's documents? (LLM08)

In multi-tenant or role-based apps, retrieval must apply the same access controls as the rest of your system. Test with two accounts that should not see each other's data.

3. Is there anything secret in the system prompt? (LLM07)

Assume the system prompt will be extracted eventually. It must never contain API keys, passwords, internal URLs or business rules you cannot afford to have published. Keep secrets and authorisation in code.

4. Where do prompts and responses go, and for how long? (LLM02, LLM03)

Know your model provider's data retention and usage terms, where processing happens and who can access logs. For Indian personal data, align this with your DPDP Act obligations.

Jailbreaks and injection

5. What untrusted content does the model read? (LLM01)

Emails, uploaded files, web pages, tickets and product reviews can all carry hidden instructions. The OWASP page on prompt injection notes that no foolproof prevention method is known, so the design must limit what a manipulated model can do.

6. Have we run adversarial tests, not just functional ones? (LLM01, LLM09)

Try direct jailbreaks, hidden instructions in documents, encoded and multilingual payloads, and requests for prohibited content. Re-run after every model or prompt change.

7. Is model output treated as untrusted before it is used? (LLM05)

If output is rendered as HTML, inserted into SQL, used in a shell command or passed to another API, it needs the same validation and encoding as any user input. Deterministic checks catch what clever wording cannot.

Tool permissions and agency

8. What can the model actually do? (LLM06)

List every tool, function and API it can call, and the permissions behind each. OWASP's excessive agency guidance warns against tools with more functionality, permissions or autonomy than the task needs.

9. Do actions run with the user's permissions or a powerful service account? (LLM06)

Prefer acting in the user's own security context with minimum scopes. A shared admin identity turns any successful injection into an admin-level incident.

10. Which actions need a human to approve? (LLM06)

Payments, deletions, external communications, permission changes and anything irreversible should require approval, enforced in code, not in the prompt.

Supply chain and data integrity

11. Do we know where our models, datasets and plugins come from? (LLM03, LLM04)

Keep an inventory of models, versions, fine-tuning data, embedding models and third-party plugins. Pin versions, review licences, and control who can change the knowledge base the system retrieves from.

Logging and monitoring

12. Can we reconstruct what happened in any conversation? (LLM01, LLM06, LLM09)

Log prompts, retrieved content, tool calls and outputs, with user and time, in a store the application cannot alter. Mask sensitive data in logs. The UK NCSC recommends this kind of logging so that suspicious activity, including repeated failed tool calls, can be detected; see its post "Prompt injection is not SQL injection".

13. Who is alerted when something looks wrong, and what do they do? (all)

Define alerts for unusual volume, high-risk tool use, policy violations and cost spikes, plus a short runbook and a switch to disable AI features quickly. Include AI incidents in your normal incident response and reporting process.

Abuse limits and cost

14. What stops one user from running up the bill or taking the service down? (LLM10)

OWASP's unbounded consumption guidance recommends input size limits, rate limiting and quotas, timeouts and monitoring for unusual usage. Set per-user, per-key and daily caps, with graceful degradation when they are hit.

Accuracy and accountability

15. What happens when the model is confidently wrong? (LLM09)

Ground answers in approved sources, show citations, let the system say "I don't know", and require human review for high-stakes output such as prices, policy commitments, legal or medical content. Tell users when they are interacting with AI.

A simple launch gate

AreaMinimum before launch
DataData map done; no secrets in prompts; retrieval access controls tested
InjectionAdversarial test round completed and findings fixed or accepted in writing
OutputOutput validated and encoded before use in other systems
AgencyLeast-privilege tools; approvals on irreversible actions
Supply chainModel and plugin inventory with versions
MonitoringFull logs, alerts, runbook and kill switch
AbuseRate limits, quotas and spend caps
AccuracyGrounding, citations and human review where stakes are high

After launch

LLM security is not a one-time sign-off. Models are updated, prompts change, new tools are connected and attackers learn. Re-run your adversarial tests after every significant change, review logs weekly in the first months, and schedule an independent test at least once a year or before major releases.

How Cehpoint can help

Our AI red teaming and LLM security testing service works through these questions against your actual application, covering prompt injection, jailbreaks, data exfiltration, tool misuse and abuse limits, and reports findings against the OWASP LLM Top 10 so your developers know what to fix. For the production side, our LLMOps and AI observability service provides the logging, alerting and cost monitoring the checklist calls for. Where the wider application also needs testing, combine it with our VAPT services.

Frequently asked questions

What is the OWASP Top 10 for LLM Applications?

It is a list of the most important security risks for applications built on large language models, maintained by the OWASP GenAI Security Project. The 2025 edition covers prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption.

Is a normal penetration test enough for an AI application?

A standard penetration test covers the web application, APIs and infrastructure, which still matter. It usually does not cover prompt injection, jailbreaks, data leakage through retrieval or excessive agency in tools. Combine a conventional VAPT with AI-specific red teaming that treats the model as an untrusted component and tests what it can be manipulated into doing.

Can we put API keys or business rules in the system prompt?

Do not put secrets in the system prompt. OWASP lists system prompt leakage as a top risk, and attackers often extract system prompts through clever questioning. Keep API keys, credentials and authorisation logic in application code and downstream systems, and assume anything in the prompt may eventually become public.

How often should we security-test an LLM application?

Test before launch, after every significant change to the model, prompts, tools or data sources, and at least once a year independently. AI behaviour can shift when a provider updates a model even if your code is unchanged, so keep an automated adversarial test set that runs regularly alongside periodic expert testing.

Need help with this?

Tell us what you are dealing with. We reply with a plain plan, and where it makes sense, a working demo before you commit.