SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement

WARNING: THIS PAPER CONTAINS RED-TEAMING RELATED CONTENT.LLM-based agent systems increasingly rely on openregistry skills, which are difficult to vet and expose agents to greater attack risks.These attacks mainly take two forms.Malicious injection attacks embed explicit harmful instructions into skills and are readily caught by auditing.Direct vulnerability attacks target latent flaws in non-malicious skills; however, directly crafted attack prompts often lead to unstable or unverified outcomes.We introduce Skil-lAttack, an automated red-teaming framework that systematically explores executable attack paths from latent vulnerabilities without modifying skill files.Evaluated across 11 target LLMs, SkillAttack substantially outperforms baselines, showing that well-intended skills can pose serious risks in realistic agent interactions.Our evaluation also yields trajectory-level attack traces with verified risk labels, providing a foundation for defense.We also release Skill-RiskBench, a benchmark for skill risk analysis, and train a lightweight pre-execution risk guard that predicts risky invocations from the user request and skill file.The guard outperforms frontier LLM judges and existing safety guard models, highlighting the value of trajectory-grounded supervision for proactive skill defense.