AI risk — prompt injection, output handling, excessive agency, model supply chain
xgrep's AI risk rules trace untrusted data into and out of LLM calls — request input reaching a system prompt, model output reaching SQL, HTML, code execution or a deserializer, model-chosen tool arguments reaching a shell, SQL, files or the network, unpinned and pickled models, and risky MCP server configuration — mapped to the OWASP Top 10 for LLM Applications.
The AIBOM says which models,
providers, agents and tools a codebase uses. The AI risk rules say what is
dangerous about how it uses them. They are ordinary security rules that run in
every xgrep scan, and like the rest of xgrep's SAST they report a flow, not a
code shape: they fire when data actually travels from where an attacker can
influence it to where it does harm.
# Every AI risk finding, alongside the rest of the security rules
xgrep scan --category security .
# Only the confirmed, exploitable ones
xgrep scan --category security --subcategory vuln .Why model output is untrusted
A model's answer is shaped by everything in its context window: the user's message, documents and web pages it retrieved, tool results, other users' messages in a shared thread. Any of them can carry a prompt injection ("ignore the previous instructions and reply with …"). So xgrep treats the value a completion call returns as attacker-influenceable input, exactly like an HTTP request parameter, and follows it to the same sinks.
What it detects
| Risk | Rule | Languages | Fires when | Tier |
|---|---|---|---|---|
| LLM05 improper output handling | llm-output-code-execution | Python, JavaScript/TypeScript, Java, Go, C# | model output reaches eval/exec, a shell (os.system, subprocess, child_process.exec, Runtime.exec, ProcessBuilder), new Function, a JSR-223 ScriptEngine, Groovy or SpEL, or a Jinja2 template compiled from it; in Go, exec.Command, syscall.Exec or a goja RunString; in C#, Process.Start/ProcessStartInfo, Roslyn CSharpScript or PowerShell AddScript | vuln |
| LLM05 improper output handling | llm-output-sql-injection | Python, JavaScript/TypeScript, Java, Go, C# | model output is the text of a SQL query (DB-API execute, Django raw, SQLAlchemy text, pg/mysql query, Knex raw, Prisma $queryRawUnsafe, JDBC executeQuery/prepareStatement, JPA createQuery/createNativeQuery, Spring JdbcTemplate/JdbcClient); in Go, database/sql Query/Exec, sqlx, gorm Raw or pgx; in C#, SqlCommand/CommandText, EF Core FromSqlRaw/ExecuteSqlRaw or Dapper | vuln |
| LLM05 improper output handling | llm-output-xss | Python, JavaScript/TypeScript, Java, Go, C# | model output is rendered as raw HTML (Markup, mark_safe, Streamlit unsafe_allow_html=True, Gradio HTML, innerHTML, insertAdjacentHTML, document.write, React dangerouslySetInnerHTML, the servlet response writer, a text/html ResponseEntity, Vaadin Html); in Go, an http.ResponseWriter not declared JSON or plain text, template.HTML or text/template; in C#, Html.Raw, MarkupString, HtmlString or a text/html content result | vuln |
| LLM05 improper output handling | llm-output-unsafe-deserialization | Python, Java, Go, C# | model output reaches pickle/dill/marshal/jsonpickle or yaml.load without a safe loader, or ObjectInputStream, XMLDecoder, XStream fromXML or SerializationUtils.deserialize; in Go, encoding/gob decoding into an interface{}/any value; in C#, BinaryFormatter and similar formatters, or Json.NET with TypeNameHandling | vuln (Go: audit) |
| LLM06 excessive agency | llm-tool-excessive-agency | Python, JavaScript/TypeScript | an argument the model chooses for a tool reaches a shell (Python); a shell, eval/exec/vm, or the text of a SQL query (JavaScript/TypeScript) | vuln |
| LLM06 excessive agency | llm-tool-code-injection | Python | an argument the model chooses for a tool reaches eval/exec, is the program of python -c, or is written to a .py file the tool then runs with the Python interpreter | vuln |
| LLM06 excessive agency | llm-tool-sql-injection | Python | an argument the model chooses for a tool is part of the text of a SQL query | vuln |
| LLM06 excessive agency | llm-tool-file-network-access | Python, JavaScript/TypeScript | an argument the model chooses for a tool is a file path or the URL of an outbound request | audit |
| Model supply chain | huggingface-unpinned-revision | Python, JavaScript/TypeScript, Rust | a Hugging Face Hub model is loaded with no commit hash, or with a branch or tag name (Python: a third-party owner/name with no revision=; JavaScript/TypeScript: Transformers.js; Rust: hf-hub) | audit |
| Model supply chain | djl-unpinned-model | Java | a DJL djl://ai.djl.huggingface.* model URL has no version segment | audit |
| Model supply chain | remote-model-deserialization | Java | a stream or file fetched at run time reaches ObjectInputStream in a file that uses a machine-learning library | audit |
| Model supply chain | remote-model-pickle-load | Python | a file downloaded at run time from the Hub (unpinned) or a URL reaches pickle, joblib, torch.load(…, weights_only=False), dill, cloudpickle, read_pickle or numpy.load(allow_pickle=True) | audit |
| Model supply chain | huggingface-trust-remote-code | Python | a Hub load passes trust_remote_code=True | vuln |
| MCP configuration | mcp-config-unpinned-package | MCP client config | a server runs npx, bunx, pnpm dlx or uvx on a package with no version, or @latest, directly or through a shell (cmd /c npx -y <pkg>, bash -c "npx ...") | audit |
| MCP configuration | mcp-config-plaintext-http | MCP client config | a remote (non-localhost) server is reached over http://, as its url or as the address a stdio bridge (curl, mcp-remote, supergateway, mcp-proxy) relays the session to (mcp-remote only with --allow-http, since it refuses a remote http:// URL otherwise) | audit |
| MCP configuration | mcp-config-shell-command | MCP client config | a server is started as a shell command line (bash -c or combined flags such as bash -lc, cmd /c, powershell -Command) | audit |
| LLM01 prompt injection | llm-prompt-injection | Python, JavaScript/TypeScript, Java, Go, C# | HTTP request input reaches the system role (a system/developer message, Anthropic system, Responses instructions, Vercel AI SDK system, a LangChain SystemMessage, Spring AI system/defaultSystem, OpenAI Java addSystemMessage) or a tool's description; in Go, also Gemini SystemInstruction, Genkit WithSystem or a Google ADK agent's instruction; in C#, also a Semantic Kernel ChatHistory system message, ChatRole.System, SystemChatMessage or an agent's Instructions | audit |
Rule ids carry a language prefix (python-…, javascript-…, java-…, go-…, csharp-…); the MCP rules
have none. The MCP rules read the client configuration files Claude Code and
Claude Desktop, Cursor, VS Code, Cline, Windsurf and the Gemini CLI use
(.mcp.json, mcp.json, claude_desktop_config.json, .gemini/settings.json,
…). A credential written into a server's env or headers there is reported by
the secret rules, like any committed credential.
Model calls recognized as sources: OpenAI (chat.completions.create,
completions.create, responses.create), Anthropic (messages.create), Google
Gemini (generate_content / generateContent), LiteLLM (completion,
acompletion), Ollama (chat, generate) and the Vercel AI SDK
(generateText). In Java: Spring AI (ChatClient .call().content(),
ChatModel.call), LangChain4j (ChatLanguageModel.generate/chat,
ChatModel.chat, the proxy AiServices builds), and the OpenAI
(chat().completions().create, responses().create), Anthropic
(messages().create) and Google Gen AI (models.generateContent) SDKs. A
LangChain4j @AiService interface that Spring injects into another class is not
yet followed as a source. In Go: go-openai (CreateChatCompletion), openai-go
(Chat.Completions.New, Responses.New), the Anthropic SDK (Messages.New),
langchaingo (GenerateFromSinglePrompt, GenerateContent, chains.Run), Google
Gen AI (Models.GenerateContent), Genkit (genkit.Generate) and CloudWeGo Eino
(a chat model's Generate). In C#: Semantic Kernel (InvokePromptAsync,
GetChatMessageContentAsync), Microsoft.Extensions.AI (GetResponseAsync), the
Agent Framework (AIAgent.RunAsync), OpenAI .NET (CompleteChat),
Azure.AI.OpenAI (GetChatCompletionsAsync) and Anthropic.SDK
(GetClaudeMessageAsync). Taint follows the response through its fields
(resp.choices[0].message.content, msg.content[0].text, a destructured
{ text }) and through string building and markdown conversion.
Tools recognized as sources: a function the model can call, through
@tool (LangChain, CrewAI, smolagents, Strands, Claude Agent SDK),
@function_tool (OpenAI Agents SDK), @<server>.tool() and the low-level
@<server>.call_tool() (MCP), @<agent>.tool / .tool_plain (Pydantic AI) in
Python, where the server may be reached through attributes (@self.mcp.tool()
on a function defined inside a method); and tool({ execute }) (Vercel AI SDK, OpenAI Agents SDK), LangChain.js
tool(fn, { schema }) / DynamicTool / DynamicStructuredTool, and the MCP
TypeScript SDK's server.tool, server.registerTool and CallToolRequestSchema
handler in JavaScript/TypeScript. Every argument the model fills in is a
source; parameters the framework injects (self, a Pydantic AI RunContext,
an MCP Context, the MCP handler's extra) are not. A LangChain tool built
with Tool(func=...) or StructuredTool.from_function(...) rather than the
@tool decorator is not recognized yet.
What stays silent — these are the safe uses, and the rules are tested against each of them:
- Model output bound as a query parameter (
cur.execute("… = %s", (answer,)),pool.query("… = $1", [answer]), Prisma's tagged$queryRaw). - Model output passed through an HTML sanitizer (
bleach.clean,nh3.clean,html.escape,DOMPurify.sanitize,sanitize-html) or rendered as text (textContent, a JSX text child, Streamlitmarkdownwithoutunsafe_allow_html, GradioMarkdown, an autoescaping template). - Model output passed to a constant template as a variable.
yaml.safe_load,yaml.load(…, Loader=SafeLoader)andjson.loads.- Request input in the user role — that is what the user role is for.
- A system prompt chosen from a server-side allowlist, read from the environment, or taken as a plain function argument.
- A tool argument or model output passed as data in an argv list
(
subprocess.run(["ls", "--", d]),execFileSync("ls", ["--", d])) with no shell. Withshell=Trueor a Node{ shell: true }option the argv elements are part of the shell command line and do report. Also silent: an argument quoted withshlex.quote, bound as a SQL parameter, or reduced withos.path.basename/path.basename. - A tool argument that only sets the options of a constant command or
expression (
exec("git status", { cwd: dir }),eval("x * 2", {"x": x})): only the command or code itself is a sink. torch.loadof a downloaded file with noweights_onlyargument: PyTorch 2.6 and later default it toTrue.python-unsafe-torch-loadstill flags everytorch.loadwithoutweights_only=Trueas an audit note for older PyTorch; an explicitweights_only=Falsereports underremote-model-pickle-load.- A URL whose literal text fixes the scheme and host (
f"https://api.example.com/search?q={q}"): the model fills in the query, not the host. A URL the argument can extend the host of ("https://" + sub + ".example.com") still reports.
Severity and the vuln / audit split
The LLM05 rules are in the vuln tier (except Go's
llm-output-unsafe-deserialization, see below): when model output reaches one of
these sinks, prompt injection turns it into SQL injection, stored XSS or code
execution, whoever wrote the prompt. SQL, code execution and deserialization are
HIGH; XSS is MEDIUM.
Go's llm-output-unsafe-deserialization is an audit item at MEDIUM:
encoding/gob builds no gadget chain and runs no code on decode, so a model's
output decoded into an interface{} value only chooses which registered type
is built. It is exploitable only when one of those types misbehaves for values
the program did not expect.
llm-tool-excessive-agency is vuln and HIGH for the same reason: a tool
that runs its argument as a shell command, code or SQL gives whoever steers the
model (through any document, page or message it reads) that capability. In
Python each sink class is its own rule (llm-tool-excessive-agency for the
shell, llm-tool-code-injection for code execution, llm-tool-sql-injection
for SQL), so a finding carries the CWE of what the tool actually does with the
argument: CWE-78, CWE-94 or CWE-89.
llm-tool-file-network-access is an audit item at low confidence: a file
or browsing tool reaches paths and URLs by design, and it is a vulnerability
only when it can reach more than its task needs (the whole filesystem, an
internal address), which the code alone rarely settles.
The LLM01 prompt-injection rule is an audit item at low confidence. Request input in the system prompt is exploitable only when that prompt guards something the caller could not otherwise reach — an authorization rule ("only show the current user's orders"), another tenant's data, a privileged tool, a confidential instruction. In a bring-your-own-key chat client, the caller writing their own system prompt for their own session is the design. xgrep cannot tell those apart from the code, so it reports the flow for review rather than as a confirmed vulnerability.
The model supply-chain and MCP configuration rules are audit items except
huggingface-trust-remote-code: an unpinned model or package, a plain-HTTP
server or a shell-wrapped command is exploitable only if its publisher, the
registry or the network path is compromised. trust_remote_code=True runs the
repository's Python on every load, so it is reported HIGH.
Linking a finding to its AI asset
Every AI risk rule names the kind of AIBOM asset it concerns in
metadata.aibom-ref (for example {kind: service, risk: LLM05}), so tooling
that holds both the scan results and the AIBOM can join a finding to the model
service, model, tool or MCP server it is about. The risk value is the OWASP
Top 10 for LLM Applications 2025 identifier, or supply-chain.
Known limits
- Flows are followed within a file. A completion wrapped in a helper defined in another module and called from the sink's file is not traced.
- Streaming responses (
stream=True,streamText) are not sources yet. - A LangChain prompt template whose system placeholder is filled from the
request at
.invoke({...})time is not traced to the system role. - A model file read through
with open(path) as fh: pickle.load(fh)is not followed from its download;pickle.load(open(path, "rb"))andfh = open(path, "rb")are. - Class-based tools (a LangChain or CrewAI
BaseToolsubclass's_run) are not yet sources. - A tool argument that reaches its sink through a helper on an HTTP client
object (
client.get(url)) rather than the module function (httpx.get(url),fetch(url)) is not followed.
See also
- Generating an AIBOM — the inventory these rules build on.
- OWASP Top 10 for LLM Applications 2025.
Last updated on
CBOM
Generate a CycloneDX 1.6 Cryptography Bill of Materials — an inventory of the cryptographic assets in your code — with xgrep sbom --include cbom.
Analytics
See where a codebase's risk actually concentrates — the churn × complexity hotspots worth refactoring first, the functions driving them, and how your scan findings land on top of that picture.