Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning

Abstract
Importance Sampling (IS) is a widely used building block for a large variety of off-policy estimation and learning algorithms. However, empirical and theoretical studies have progressively shown that vanilla IS leads to poor estimations whenever the behavioral and target policies are too dissimilar. In this paper, we analyze the theoretical properties of the IS estimator by deriving a novel anticoncentration bound that formalizes the intuition behind its undesired behavior. Then, we propose a new class of IS transformations, based on the notion of power mean. To the best of our knowledge, the resulting estimator is the first to achieve, under certain conditions, two key properties: (i) it displays a subgaussian concentration rate; (ii) it preserves the differentiability in the target distribution. Finally, we provide numerical simulations on both synthetic examples and contextual bandits, in comparison with off-policy evaluation and learning baselines.

[Link] [Poster] [Slides] [Code] [BibTeX]

 @incollection{metelli2021subgaussian,
    author = "Metelli, Alberto Maria and Russo, Alessio and Restelli, Marcello",
    title = "Subgaussian and Differentiable Importance Sampling for Off-Policy Evaluation and Learning",
    booktitle = "Advances in Neural Information Processing Systems (NeurIPS)",
    year = "2021",
    pages = "8119--8132",
    url = "https://proceedings.neurips.cc/paper/2021/hash/4476b929e30dd0c4e8bdbcc82c6ba23a-Abstract.html"
}