Trust-Gated Behavioral Auditing: Detecting Reward Hacking in Reinforcement Learning Agents
Ajay Bandiwaddar
阅读操作
确认中在文库中上传 PDF 后可生成中文音频讲解。
摘要与影响
Reinforcement learning agents are evaluated almost exclusively by cumulative reward — a proxy that Skalse et al. [NeurIPS 2022] prove is mathematically hackable for any non-constant objective. Gao et al. [ICML 2023] document the empirical consequence: proxy reward rises while true performance peaks and falls, a divergence invisible to any system tracking only the reward curve. No standard evaluation tool diagnoses this gap post-hoc, per-agent, without modifying the training pipeline. We introduce LearnLens, a Python package computing a Learning Quality Score (LQS) — a composite behavioral metric decomposing agent behavior into four probes grounded in the Goodhart taxonomy: Generalization (G), Consistency (C), Hack Index (H), and Reasoning Alignment (R), combined as LQS = sqrt(G x C) x (1 - sqrt(H)) + 0.15 x R x (1 - sqrt(H)), capped at 1.0. In a controlled three-agent experiment, LQS correctly ranked agents by true quality where cumulative reward did not. In a GRPO experiment using Qwen2.5-3B-Instruct on a T4 GPU over 500 steps, an LQS-inspired penalty reduced Hack Index from 1.00 to 0.00 and raised LQS from 0.000 to 0.848, while cumulative reward increased only 46.5%. LearnLens is pip-installable (pip install learnlens-rl), compatible with Gymnasium, Stable-Baselines3, and the OpenEnv ecosystem, and fully open-source.
逐年被引趋势
暂无年度引用数据
关键指标
同类平均 = 1
同领域 · 同年份 · 同类型
Google Scholar 与 OpenAlex 的被引统计范围不同,数值存在差异属正常。
AI 辅助阅读
依据:摘要
可就本文提问;依据不足时会说明。
学术脉络
学科主题
计算机 / AIReinforcement Learning in Robotics
Adversarial Robustness in Machine Learning · Artificial Intelligence in Games