The LLM API is still an API

An AI feature almost never reaches the user as a bare model, but as an HTTP endpoint: a POST /chat that accepts a prompt, passes it on to a language model, and returns the response. That means all the rules that apply to any API apply here first. Anyone who only thinks of prompt injection here is overlooking the basics.

Concretely, that means: the LLM endpoint needs authentication and authorization like any other. An object-level check (BOLA) decides whether a user is even allowed to access the document, conversation, or record they reference via the prompt. Without this check, the model becomes a bypass path: the attacker doesn't query the protected resource directly, but asks the AI for it instead.

  • Rate limiting and quotas: LLM inference is compute- and cost-intensive. Lack of limits leads to denial of service and direct financial losses. OWASP lists this as its own risk, LLM10:2025 Unbounded Consumption, which among other things covers variable-length input floods and model theft.
  • Input validation: even if the input field is free text, length, format, and the number of attachments still need to be checked.
  • Error handling and configuration: stack traces, model names, or system prompts must not leak through into responses or error messages (cf. API8:2023 Security Misconfiguration).

The model-specific risks therefore come in addition to classic API protection, not instead of it.

IDRiskCore of the DangerKey Countermeasure
LLM01Prompt InjectionInputs (direct or indirect) unintentionally alter model behaviorRestrict behavior via system prompt, flag and separate external content, least privilege
LLM02Sensitive Information DisclosurePII, secrets, or credentials end up in the model outputData scrubbing/masking before processing, access control, no training on inputs
LLM05Improper Output HandlingModel outputs passed on unchecked lead to XSS, SSRF, RCETreat model outputs like user input, encode according to context
LLM06Excessive AgencyToo many tools, permissions, or autonomy allow harmful actionsMinimize tools, backend authorization (complete mediation), human-in-the-loop
LLM10Unbounded ConsumptionUncontrolled inference leads to DoS, cost explosion, model theftRate limiting, quotas, and spend limits, e.g. at the AI gateway

OWASP Top 10 for LLM Applications (2025): core risks for API operators

OWASP Top 10 for LLM Applications: the three core risks

The OWASP Gen AI Security Project maintains the de facto standard for model-specific risks with the Top 10 for LLM Applications (2025 edition). Three categories are especially relevant for API operators.

LLM01:2025 Prompt Injection

Prompt injection occurs when inputs unintentionally alter the model's behavior or output. OWASP distinguishes between direct injection (the user manipulates the prompt themselves) and indirect injection (the model processes manipulated content from external sources such as websites, documents, or tool responses). Because generative models work stochastically, OWASP states there is no foolproof prevention. Effective measures include restricting behavior via system prompt, filtering inputs and outputs, clearly flagging external content and separating it from the user prompt, and enforcing least privilege for all model access.

LLM02:2025 Sensitive Information Disclosure

The model can leak personal data (PII), trade secrets, credentials, or proprietary algorithms through its outputs. Causes include insufficient data scrubbing, missing access control on connected data sources, or training data that contains sensitive content. OWASP recommends data scrubbing (cleansing, masking) before any processing, strict access controls, and clear terms of use. Important: a mere instruction in the system prompt not to disclose certain data can be bypassed via prompt injection and is therefore not a reliable control.

LLM06:2025 Excessive Agency

As soon as a model is allowed to call functions or operate tools, plugins, and extensions, agency arises. Excessive Agency is the vulnerability that enables harmful actions from unexpected, ambiguous, or manipulated model outputs. OWASP names three root causes: excessive functionality (the tool can do more than necessary, e.g. delete instead of only read), excessive permissions (the database account used has UPDATE/DELETE instead of only SELECT), and excessive autonomy (actions without confirmation). Countermeasures are reducing the tools and their functions, backend permissions following least privilege, execution in the context of the respective user (e.g. OAuth with minimal scope), and a human-in-the-loop for high-impact actions.

Tool use and data exfiltration: when the model acts

The most dangerous combination arises when indirect prompt injection meets excessive agency. OWASP illustrates this with the example of an assistant with mailbox access: a crafted incoming email instructs the model to search the mailbox for sensitive information and forward it to an attacker address. The user never entered anything malicious - the attack is embedded in the processed content.

This leads to a clear separation of responsibilities: the model decides what should be done, but the authorization over whether it may be done does not belong in the prompt. OWASP frames this as the principle of complete mediation: authorization is enforced in the downstream system, not decided by the model. In practice, that means:

  • Tool calls run through dedicated functions checked deterministically in code, rather than through open-ended capabilities such as run_shell or fetch_url.
  • Every tool call carries the identity and scope of the acting user, not a generic high-privilege account.
  • Model outputs that are processed downstream are treated like user input - otherwise, per LLM05:2025 Improper Output Handling, XSS, SSRF, or even remote code execution are risks.
  • High-impact actions (deleting, sending, payments) require human approval.

The data exfiltration path is not just the response to the user, but every channel a tool can open: outbound HTTP requests, emails, database write access.

AI code generators expand the attack surface

A separate risk path emerges away from runtime: in development itself. AI code generators and coding agents quickly produce new endpoints, helper routes, debug interfaces, and versions. What goes to production and what remains just a draft is often not cleanly documented. This creates phantom or shadow endpoints: routes that are reachable but appear in no inventory and no API specification.

OWASP describes exactly this pattern as API9:2023 Improper Inventory Management. Attackers typically gain access via old API versions or forgotten, unpatched endpoints with weaker security requirements. According to OWASP, an API has a documentation blind spot when it is unclear which version is running, whether documentation exists, and whether a decommissioning plan exists. AI generation accelerates the growth of exactly this unclear surface.

The countermeasures are classic but gain weight: a maintained endpoint inventory, automated API discovery against actual traffic (not just against the documentation), versioning and retirement plans, and reconciling generated routes with the approved specification before they go live. Anyone using AI in the build therefore needs more inventory discipline, not less.

AI gateways: a central control point for LLM traffic

Just as a classic API gateway sits between clients and backends, an AI gateway inserts itself between application and model provider. It bundles cross-cutting tasks for LLM traffic at a single point. Kong describes its AI Gateway as a control and governance layer that routes to different LLM providers via a provider-agnostic API, centrally manages credentials, and directs requests by cost, latency, or availability. Cloudflare describes its AI Gateway as a visibility and control layer with analytics and logging, caching, rate limiting, retries, and model fallback.

The security-relevant functions above all are:

  • Central key management: provider keys live in the gateway instead of being scattered across applications, which reduces key leakage.
  • Rate limiting and spend limits: a direct countermeasure against unbounded consumption.
  • Observability: prompts, token consumption, and costs are logged and thus auditable.
  • Guardrails and moderation: inputs and outputs can be checked against policies. Mistral's moderation, for example, includes a dedicated jailbreaking category and can use custom guardrails to block inputs before they reach the model (HTTP 403).

However, an AI gateway does not replace any of the OWASP measures. It is an enforcement point - per-user authorization, clean tool design, and output handling remain the application's responsibility. And as with any gateway: if a managed third-party SaaS sees the entire prompt traffic, including sensitive payloads, that is a deliberate sovereignty decision.

Implemented with sovereignty: the Venedy approach

Venedy processes its own AI analyses via an EU AI provider and runs on European infrastructure. Secrets are masked before every AI request, and the submitted data does not flow into model training. This addresses exactly the risks from LLM02:2025 Sensitive Information Disclosure: data scrubbing before processing, and a provider where inputs do not become training material.

For organizations that ship their own LLM features, the approach can be generalized:

  1. Secure the LLM endpoint like any API: authentication, object- and function-level authorization, rate limiting, clean error handling.
  2. Scrub or mask sensitive data before the model request and give the provider only what's strictly necessary.
  3. Choose an EU provider that doesn't train on customer data if data location and legal jurisdiction are relevant - this fits within the framework of DORA, NIS2, and GDPR.
  4. Scope tools restrictively, execute them in the user's context, and require human approval for impactful actions (excessive agency).
  5. Maintain the endpoint inventory, especially when AI code generators are part of the build.

Orientation is provided by the OWASP Gen AI Security Project, NIST AI 600-1 (Generative AI Profile to the AI Risk Management Framework, July 2024), and the BSI publication on generative AI models, which summarizes risks and countermeasures for companies and authorities.

Sources

Keep reading
API5Broken Function Level Authorization Attacks & RisksImproper Inventory Management SovereigntySovereignty in a Security Context