Policy space identification in configurable environments

Abstract
We study the problem of identifying the policy space available to an agent in a learning process, having access to a set of demonstrations generated by the agent playing the optimal policy in the considered space. We introduce an approach based on frequentist statistical testing to identify the set of policy parameters that the agent can control, within a larger parametric policy space. After presenting two identification rules (combinatorial and simplified), applicable under different assumptions on the policy space, we provide a probabilistic analysis of the simplified one in the case of linear policies belonging to the exponential family. To improve the performance of our identification rules, we make use of the recently introduced framework of the Configurable Markov Decision Processes, exploiting the opportunity of configuring the environment to induce the agent to reveal which parameters it can control. Finally, we provide an empirical evaluation, on both discrete and continuous domains, to prove the effectiveness of our identification rules.

[Link] [Slides] [BibTeX]

 @article{metelli2019policy,
    author = "Metelli, Alberto Maria and Manneschi, Guglielmo and Restelli, Marcello",
    title = "Policy space identification in configurable environments",
    journal = "Machine Learning",
    volume = "111",
    number = "6",
    pages = "2093--2145",
    year = "2022",
    url = "https://doi.org/10.1007/s10994-021-06033-3",
    doi = "10.1007/s10994-021-06033-3"
}