Orateur
Description
When facing contextual multi-stage optimization problems, training combinatorial optimization-enriched machine learning pipelines (ML-CO-pipelines) to date either requires imitating expert solutions or utilizing unstructured learning algorithms. While the former restricts the use of ML-CO-pipelines to problems with traceable offline solutions and relatively homogenous state spaces, the latter fails to account for combinatorial action spaces, which can destabilize training.
To mitigate the respective drawbacks, we introduce structured reinforcement learning (SRL), enabling the stable training of ML-CO pipelines using only collected experience. In the core of its training process, SRL generates and evaluates several actions by perturbing the ML model’s output and subsequently performs an update step towards the best actions using a Fenchel-Young loss.
We test SRL in three static and three dynamic environments, representing various industrial applications. We find SRL to substantially outperform structured imitation learning and unstructured RL, attributing to its enhanced exploration of the combinatorial state-action space and to its improved training stability.
