Safe Policy Iteration: A Monotonically Improving Approximate Policy Iteration Approach

Abstract
This paper presents a study of the policy improvement step that can be usefully exploited by approximate policy–iteration algorithms. When either the policy evaluation step or the policy improvement step returns an approximated result, the sequence of policies produced by policy iteration may not be monotonically increasing, and oscillations may occur. To address this issue, we consider safe policy improvements, i.e., at each iteration we search for a policy that maximizes a lower bound to the policy improvement w.r.t. the current policy, until no improving policy can be found. We propose three safe policy–iteration schemata that differ in the way the next policy is chosen w.r.t. the estimated greedy policy. Besides being theoretically derived and discussed, the proposed algorithms are empirically evaluated and compared on some chain-walk domains, the prison domain and on the Blackjack card game.

[Link] [BibTeX]

 @article{metelli2018safe,
    author = "Metelli, Alberto Maria and Pirotta, Matteo and Calandriello, Daniele and Restelli, Marcello",
    title = "Safe Policy Iteration: A Monotonically Improving Approximate Policy Iteration Approach",
    journal = "Journal of Machine Learning Research",
    year = "2021",
    volume = "22",
    number = "97",
    pages = "1-83",
    url = "http://jmlr.org/papers/v22/19-707.html"
}