Skip to Content
Code Scanning

AI risk — prompt injection, output handling, excessive agency, model supply chain

xgrep's AI risk rules trace untrusted data into and out of LLM calls — request input reaching a system prompt, model output reaching SQL, HTML, code execution or a deserializer, model-chosen tool arguments reaching a shell, SQL, files or the network, unpinned and pickled models, and risky MCP server configuration — mapped to the OWASP Top 10 for LLM Applications.

The AIBOM says which models, providers, agents and tools a codebase uses. The AI risk rules say what is dangerous about how it uses them. They are ordinary security rules that run in every xgrep scan, and like the rest of xgrep's SAST they report a flow, not a code shape: they fire when data actually travels from where an attacker can influence it to where it does harm.

# Every AI risk finding, alongside the rest of the security rules
xgrep scan --category security .

# Only the confirmed, exploitable ones
xgrep scan --category security --subcategory vuln .

Why model output is untrusted

A model's answer is shaped by everything in its context window: the user's message, documents and web pages it retrieved, tool results, other users' messages in a shared thread. Any of them can carry a prompt injection ("ignore the previous instructions and reply with …"). So xgrep treats the value a completion call returns as attacker-influenceable input, exactly like an HTTP request parameter, and follows it to the same sinks.

What it detects

RiskRuleLanguagesFires whenTier
LLM05 improper output handlingllm-output-code-executionPython, JavaScript/TypeScript, Java, Go, C#model output reaches eval/exec, a shell (os.system, subprocess, child_process.exec, Runtime.exec, ProcessBuilder), new Function, a JSR-223 ScriptEngine, Groovy or SpEL, or a Jinja2 template compiled from it; in Go, exec.Command, syscall.Exec or a goja RunString; in C#, Process.Start/ProcessStartInfo, Roslyn CSharpScript or PowerShell AddScriptvuln
LLM05 improper output handlingllm-output-sql-injectionPython, JavaScript/TypeScript, Java, Go, C#model output is the text of a SQL query (DB-API execute, Django raw, SQLAlchemy text, pg/mysql query, Knex raw, Prisma $queryRawUnsafe, JDBC executeQuery/prepareStatement, JPA createQuery/createNativeQuery, Spring JdbcTemplate/JdbcClient); in Go, database/sql Query/Exec, sqlx, gorm Raw or pgx; in C#, SqlCommand/CommandText, EF Core FromSqlRaw/ExecuteSqlRaw or Dappervuln
LLM05 improper output handlingllm-output-xssPython, JavaScript/TypeScript, Java, Go, C#model output is rendered as raw HTML (Markup, mark_safe, Streamlit unsafe_allow_html=True, Gradio HTML, innerHTML, insertAdjacentHTML, document.write, React dangerouslySetInnerHTML, the servlet response writer, a text/html ResponseEntity, Vaadin Html); in Go, an http.ResponseWriter not declared JSON or plain text, template.HTML or text/template; in C#, Html.Raw, MarkupString, HtmlString or a text/html content resultvuln
LLM05 improper output handlingllm-output-unsafe-deserializationPython, Java, Go, C#model output reaches pickle/dill/marshal/jsonpickle or yaml.load without a safe loader, or ObjectInputStream, XMLDecoder, XStream fromXML or SerializationUtils.deserialize; in Go, encoding/gob decoding into an interface{}/any value; in C#, BinaryFormatter and similar formatters, or Json.NET with TypeNameHandlingvuln (Go: audit)
LLM06 excessive agencyllm-tool-excessive-agencyPython, JavaScript/TypeScriptan argument the model chooses for a tool reaches a shell (Python); a shell, eval/exec/vm, or the text of a SQL query (JavaScript/TypeScript)vuln
LLM06 excessive agencyllm-tool-code-injectionPythonan argument the model chooses for a tool reaches eval/exec, is the program of python -c, or is written to a .py file the tool then runs with the Python interpretervuln
LLM06 excessive agencyllm-tool-sql-injectionPythonan argument the model chooses for a tool is part of the text of a SQL queryvuln
LLM06 excessive agencyllm-tool-file-network-accessPython, JavaScript/TypeScriptan argument the model chooses for a tool is a file path or the URL of an outbound requestaudit
Model supply chainhuggingface-unpinned-revisionPython, JavaScript/TypeScript, Rusta Hugging Face Hub model is loaded with no commit hash, or with a branch or tag name (Python: a third-party owner/name with no revision=; JavaScript/TypeScript: Transformers.js; Rust: hf-hub)audit
Model supply chaindjl-unpinned-modelJavaa DJL djl://ai.djl.huggingface.* model URL has no version segmentaudit
Model supply chainremote-model-deserializationJavaa stream or file fetched at run time reaches ObjectInputStream in a file that uses a machine-learning libraryaudit
Model supply chainremote-model-pickle-loadPythona file downloaded at run time from the Hub (unpinned) or a URL reaches pickle, joblib, torch.load(…, weights_only=False), dill, cloudpickle, read_pickle or numpy.load(allow_pickle=True)audit
Model supply chainhuggingface-trust-remote-codePythona Hub load passes trust_remote_code=Truevuln
MCP configurationmcp-config-unpinned-packageMCP client configa server runs npx, bunx, pnpm dlx or uvx on a package with no version, or @latest, directly or through a shell (cmd /c npx -y <pkg>, bash -c "npx ...")audit
MCP configurationmcp-config-plaintext-httpMCP client configa remote (non-localhost) server is reached over http://, as its url or as the address a stdio bridge (curl, mcp-remote, supergateway, mcp-proxy) relays the session to (mcp-remote only with --allow-http, since it refuses a remote http:// URL otherwise)audit
MCP configurationmcp-config-shell-commandMCP client configa server is started as a shell command line (bash -c or combined flags such as bash -lc, cmd /c, powershell -Command)audit
LLM01 prompt injectionllm-prompt-injectionPython, JavaScript/TypeScript, Java, Go, C#HTTP request input reaches the system role (a system/developer message, Anthropic system, Responses instructions, Vercel AI SDK system, a LangChain SystemMessage, Spring AI system/defaultSystem, OpenAI Java addSystemMessage) or a tool's description; in Go, also Gemini SystemInstruction, Genkit WithSystem or a Google ADK agent's instruction; in C#, also a Semantic Kernel ChatHistory system message, ChatRole.System, SystemChatMessage or an agent's Instructionsaudit

Rule ids carry a language prefix (python-…, javascript-…, java-…, go-…, csharp-…); the MCP rules have none. The MCP rules read the client configuration files Claude Code and Claude Desktop, Cursor, VS Code, Cline, Windsurf and the Gemini CLI use (.mcp.json, mcp.json, claude_desktop_config.json, .gemini/settings.json, …). A credential written into a server's env or headers there is reported by the secret rules, like any committed credential.

Model calls recognized as sources: OpenAI (chat.completions.create, completions.create, responses.create), Anthropic (messages.create), Google Gemini (generate_content / generateContent), LiteLLM (completion, acompletion), Ollama (chat, generate) and the Vercel AI SDK (generateText). In Java: Spring AI (ChatClient .call().content(), ChatModel.call), LangChain4j (ChatLanguageModel.generate/chat, ChatModel.chat, the proxy AiServices builds), and the OpenAI (chat().completions().create, responses().create), Anthropic (messages().create) and Google Gen AI (models.generateContent) SDKs. A LangChain4j @AiService interface that Spring injects into another class is not yet followed as a source. In Go: go-openai (CreateChatCompletion), openai-go (Chat.Completions.New, Responses.New), the Anthropic SDK (Messages.New), langchaingo (GenerateFromSinglePrompt, GenerateContent, chains.Run), Google Gen AI (Models.GenerateContent), Genkit (genkit.Generate) and CloudWeGo Eino (a chat model's Generate). In C#: Semantic Kernel (InvokePromptAsync, GetChatMessageContentAsync), Microsoft.Extensions.AI (GetResponseAsync), the Agent Framework (AIAgent.RunAsync), OpenAI .NET (CompleteChat), Azure.AI.OpenAI (GetChatCompletionsAsync) and Anthropic.SDK (GetClaudeMessageAsync). Taint follows the response through its fields (resp.choices[0].message.content, msg.content[0].text, a destructured { text }) and through string building and markdown conversion.

Tools recognized as sources: a function the model can call, through @tool (LangChain, CrewAI, smolagents, Strands, Claude Agent SDK), @function_tool (OpenAI Agents SDK), @<server>.tool() and the low-level @<server>.call_tool() (MCP), @<agent>.tool / .tool_plain (Pydantic AI) in Python, where the server may be reached through attributes (@self.mcp.tool() on a function defined inside a method); and tool({ execute }) (Vercel AI SDK, OpenAI Agents SDK), LangChain.js tool(fn, { schema }) / DynamicTool / DynamicStructuredTool, and the MCP TypeScript SDK's server.tool, server.registerTool and CallToolRequestSchema handler in JavaScript/TypeScript. Every argument the model fills in is a source; parameters the framework injects (self, a Pydantic AI RunContext, an MCP Context, the MCP handler's extra) are not. A LangChain tool built with Tool(func=...) or StructuredTool.from_function(...) rather than the @tool decorator is not recognized yet.

What stays silent — these are the safe uses, and the rules are tested against each of them:

  • Model output bound as a query parameter (cur.execute("… = %s", (answer,)), pool.query("… = $1", [answer]), Prisma's tagged $queryRaw).
  • Model output passed through an HTML sanitizer (bleach.clean, nh3.clean, html.escape, DOMPurify.sanitize, sanitize-html) or rendered as text (textContent, a JSX text child, Streamlit markdown without unsafe_allow_html, Gradio Markdown, an autoescaping template).
  • Model output passed to a constant template as a variable.
  • yaml.safe_load, yaml.load(…, Loader=SafeLoader) and json.loads.
  • Request input in the user role — that is what the user role is for.
  • A system prompt chosen from a server-side allowlist, read from the environment, or taken as a plain function argument.
  • A tool argument or model output passed as data in an argv list (subprocess.run(["ls", "--", d]), execFileSync("ls", ["--", d])) with no shell. With shell=True or a Node { shell: true } option the argv elements are part of the shell command line and do report. Also silent: an argument quoted with shlex.quote, bound as a SQL parameter, or reduced with os.path.basename / path.basename.
  • A tool argument that only sets the options of a constant command or expression (exec("git status", { cwd: dir }), eval("x * 2", {"x": x})): only the command or code itself is a sink.
  • torch.load of a downloaded file with no weights_only argument: PyTorch 2.6 and later default it to True. python-unsafe-torch-load still flags every torch.load without weights_only=True as an audit note for older PyTorch; an explicit weights_only=False reports under remote-model-pickle-load.
  • A URL whose literal text fixes the scheme and host (f"https://api.example.com/search?q={q}"): the model fills in the query, not the host. A URL the argument can extend the host of ("https://" + sub + ".example.com") still reports.

Severity and the vuln / audit split

The LLM05 rules are in the vuln tier (except Go's llm-output-unsafe-deserialization, see below): when model output reaches one of these sinks, prompt injection turns it into SQL injection, stored XSS or code execution, whoever wrote the prompt. SQL, code execution and deserialization are HIGH; XSS is MEDIUM.

Go's llm-output-unsafe-deserialization is an audit item at MEDIUM: encoding/gob builds no gadget chain and runs no code on decode, so a model's output decoded into an interface{} value only chooses which registered type is built. It is exploitable only when one of those types misbehaves for values the program did not expect.

llm-tool-excessive-agency is vuln and HIGH for the same reason: a tool that runs its argument as a shell command, code or SQL gives whoever steers the model (through any document, page or message it reads) that capability. In Python each sink class is its own rule (llm-tool-excessive-agency for the shell, llm-tool-code-injection for code execution, llm-tool-sql-injection for SQL), so a finding carries the CWE of what the tool actually does with the argument: CWE-78, CWE-94 or CWE-89. llm-tool-file-network-access is an audit item at low confidence: a file or browsing tool reaches paths and URLs by design, and it is a vulnerability only when it can reach more than its task needs (the whole filesystem, an internal address), which the code alone rarely settles.

The LLM01 prompt-injection rule is an audit item at low confidence. Request input in the system prompt is exploitable only when that prompt guards something the caller could not otherwise reach — an authorization rule ("only show the current user's orders"), another tenant's data, a privileged tool, a confidential instruction. In a bring-your-own-key chat client, the caller writing their own system prompt for their own session is the design. xgrep cannot tell those apart from the code, so it reports the flow for review rather than as a confirmed vulnerability.

The model supply-chain and MCP configuration rules are audit items except huggingface-trust-remote-code: an unpinned model or package, a plain-HTTP server or a shell-wrapped command is exploitable only if its publisher, the registry or the network path is compromised. trust_remote_code=True runs the repository's Python on every load, so it is reported HIGH.

Linking a finding to its AI asset

Every AI risk rule names the kind of AIBOM asset it concerns in metadata.aibom-ref (for example {kind: service, risk: LLM05}), so tooling that holds both the scan results and the AIBOM can join a finding to the model service, model, tool or MCP server it is about. The risk value is the OWASP Top 10 for LLM Applications 2025 identifier, or supply-chain.

Known limits

  • Flows are followed within a file. A completion wrapped in a helper defined in another module and called from the sink's file is not traced.
  • Streaming responses (stream=True, streamText) are not sources yet.
  • A LangChain prompt template whose system placeholder is filled from the request at .invoke({...}) time is not traced to the system role.
  • A model file read through with open(path) as fh: pickle.load(fh) is not followed from its download; pickle.load(open(path, "rb")) and fh = open(path, "rb") are.
  • Class-based tools (a LangChain or CrewAI BaseTool subclass's _run) are not yet sources.
  • A tool argument that reaches its sink through a helper on an HTTP client object (client.get(url)) rather than the module function (httpx.get(url), fetch(url)) is not followed.

See also

Last updated on

On this page