PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

What happened

arXiv:2609.31855v1 Announce Type: new Abstract: Human demonstrations provide dense policy-level information but sometimes lack local precision. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated.

Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions.