This study compares reinforcement finetuning algorithms and parameter-update choices for Transformer-based agents using few-shot offline data. It examines multiple meta-RL environments and combines supervised and reinforcement objectives to improve adaptation. The analysis explores when reinforcement objectives add value beyond supervised finetuning, rather than assuming one adaptation strategy is uniformly effective.