On the use of the policy gradient and Hessian in inverse reinforcement learning

Abstract
Reinforcement Learning (RL) is an effective approach to solve sequential decision making problems when the environment is equipped with a reward function to evaluate the agent’s actions. However, there are several domains in which a reward function is not available and difficult to estimate. When samples of expert agents are available, Inverse Reinforcement Learning (IRL) allows recovering a reward function that explains the demonstrated behavior. Most of the classic IRL methods, in addition to expert’s demonstrations, require sampling the environment to evaluate each reward function, that, in turn, is built starting from a set of engineered features. This paper is about a novel model-free IRL approach that does not require to specify a function space where to search for the expert’s reward function. Leveraging on the fact that the policy gradient needs to be zero for an optimal policy, the algorithm generates an approximation space for the reward function, in which a reward is singled out employing a second-order criterion. After introducing our approach for finite domains, we extend it to continuous ones. The empirical results, on both finite and continuous domains, show that the reward function recovered by our algorithm allows learning policies that outperform those obtained with the true reward function, in terms of learning speed.

[Link] [BibTeX]

 @article{metelli2019ongradient,
    author = "Metelli, Alberto Maria and Pirotta, Matteo and Restelli, Marcello",
    title = "On the use of the policy gradient and Hessian in inverse reinforcement learning",
    journal = "Intelligenza Artificiale",
    volume = "14",
    number = "1",
    pages = "117--150",
    year = "2020",
    url = "https://doi.org/10.3233/IA-180011",
    doi = "10.3233/IA-180011",
    note = "Invited publication as winner of the \quotes{Premio NeoLaureati Leonardo Lesmo 2018}"
}