Distributional reinforcement learning (DRL) extends the value-based approach by approximating the full distribution over future returns instead of the mean only, providing a richer signal that leads to improved performances. Quantile Regression (QR) based methods like QR-DQN project arbitrary distributions into a parametric subset of staircase distributions by minimizing the 1-Wasserstein distance. However, due to biases in the gradients, the quantile regression loss is used instead for training, guaranteeing the same minimizer and enjoying unbiased gradients. Non-crossing constraints on the quantiles have been shown to improve the performance of QR-DQN for uncertainty-based exploration strategies. The contribution of this work is in the setting of fixed quantile levels and is twofold. First, we prove that the Cramér distance yields a projection that coincides with the 1-Wasserstein one and that, under non-crossing constraints, the squared Cramér and the quantile regression losses yield collinear gradients, shedding light on the connection between these important elements of DRL. Second, we propose a low complexity algorithm to compute the Cramér distance.
@inproceedings{Lheritier2022Cramer,
author = {Alix Lh{\'e}ritier and Nicolas Bondoux},
title = {A Cram{\'e}r Distance Perspective on Quantile Regression Based Distributional Reinforcement Learning},
booktitle = {Proceedings of the 25th International Conference on Artificial Intelligence and Statistics (AISTATS 2022)},
year = {2022},
publisher = {PMLR}
}