langsmith-evaluator

Name: langsmith-skills/langsmith-evaluator
Author: langchain-ai

Share on X Share on LinkedIn Share on Bluesky

100Critical

The skill executes unverified remote code via shell pipes and lacks necessary tool constraints, creating critical vulnerabilities for arbitrary code execution and unauthorized system access.

ThreatsCI SM CS BC S HITL

Threat Analysis

AI Agent Traps ↗Scanned 6/20/2026

Content InjectionPerceptionclean

Semantic ManipulationReasoningclean

Cognitive StateMemory & Learningclean

Behavioural ControlAction3 findings

SystemicMulti-Agent Dynamicsclean

Human-in-the-LoopHuman Overseerclean

Skill Info

Namelangchain-ai/langsmith-skills/langsmith-evaluator

Registrygithub

Version68c8bb6b4b7c

PURLpkg:github/langchain-ai/langsmith-skills@68c8bb6b4b7c?skill=langsmith-evaluator

Stars131

Installs2,500

Source

Repository ↗SKILL.md ↗

Install

This skill has been flagged as potentially malicious. Review the findings below before installing.

GitHub

2,500 installs

Skills.shDocs

npx skills add https://github.com/langchain-ai/langsmith-skills

Assessments (3)

AI Agent Traps ↗

Remote code download and execution detectedcriticalcommand executionAML.T0050AML.T0053AML.T0072LLM05:2025LLM06:2025

Remote code download and execution detected

100% confidenceline 34

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

A remote resource is downloaded with curl/wget and piped directly into a shell interpreter (optionally via sudo). The downloaded script runs with no integrity check (no checksum/signature), so a compromised server, MITM, or hijacked URL leads to arbitrary code execution. Download to a file, verify it (e.g. sha256sum -c against a pinned digest, or a signature), then execute it.criticalvulnAML.T0050AML.T0053AML.T0072LLM05:2025LLM06:2025

95% confidenceline 1

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

Credential exposure risk in documentationmediumcredential theftAML.T0055AML.T0037LLM02:2025

The documentation encourages users to set API keys as environment variables or pass them as CLI flags, which may lead to accidental exposure in shell history or process logs.

80% confidenceline 14

LANGSMITH_API_KEY=lsv2_pt_your_api_key_here

Missing allowed-tools declarationlowunauthorized tool use

Skill executes commands, writes files, or accesses the network but declares no allowed-tools, so its tool surface cannot be reviewed or constrained.

90% confidence

Undeclared network usagelowunauthorized tool use

Skill contains network access patterns (HTTP requests, socket connections) without declaring network capability in allowed-tools.

60% confidence

Missing licenseinfopolicy violation

Skill does not specify a license field. Specifying a license helps users understand usage terms.

100% confidence

Badge

Markdown

[![Mondoo Skill Check](https://mondoo.com/ai-agent-security/api/badge/github/langchain-ai/langsmith-skills/langsmith-evaluator.svg)](https://mondoo.com/ai-agent-security/skills/github/langchain-ai/langsmith-skills/langsmith-evaluator)

HTML

<a href="https://mondoo.com/ai-agent-security/skills/github/langchain-ai/langsmith-skills/langsmith-evaluator"><img src="https://mondoo.com/ai-agent-security/api/badge/github/langchain-ai/langsmith-skills/langsmith-evaluator.svg" alt="Mondoo Skill Check" /></a>

Image URL

https://mondoo.com/ai-agent-security/api/badge/github/langchain-ai/langsmith-skills/langsmith-evaluator.svg

Secure your AI agents

Skills can read files, run commands, and access credentials. Mondoo helps organizations manage the security risks of AI agent skills across their entire fleet.

Continuous scanning of skills across all registries
Policy enforcement before skills reach your agents
Integration with your existing security workflow