Plasticity is a network’s ability to keep learning from new data. In visual RL, that ability matters because the data and training targets evolve as the agent interacts with its environment.
The ICLR 2024 study examines three parts of this problem: the training data, the network’s modules and the stage of learning. Its experiments find that augmentation helps preserve plasticity, that the critic can become the main bottleneck, and that losses early in training need timely attention. Paper summary.
That observation motivates Adaptive Replay Ratio. It adjusts the frequency of data reuse according to the critic’s plasticity, protecting early learning while allowing more reuse later. The official implementation makes this training strategy available for further study.