Robust Reinforcement Learning

This is also a paradigm, a term that I first heard by reading the FormulaZero paper.

Robustness is actually really important, because if there is failure in the system, it can be very dangerous and harm people.

The goal of Robust reinforcement learning (RL) is to find a policy that optimizes the worst-case performance over an uncertainty set of MDPs.

Robustness is important because:

• Cost of failure is high
• Model is not known and created from few samples

A robust policy optimizes for the robust (worst-case) expected return objective: $J_R(\pi )={\inf }_{p\in P}{\mathbb{E}}^{p,\pi }[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_t]$

Context of Autonomous Racing

See FormulaZero paper.

Why Robustness and modelling uncertainty?

• It’s this tradeoff between safety and performance
• Well, you don’t know how your opponent is going to drive, but you want to avoid crashing
• So you maintain a larger ambiguity set to ensure a greater degree of safety while sacrificing performance against a particular opponent.