Off-policy Learning in Large-scale POMDP-based Dialogue Systems

Lucie Daubigney (1, 2) , Matthieu Geist (2) , Olivier Pietquin (2)

1 MAIA - Autonomous intelligent machine
2 IMS : Information, Multimodalité & Signal

Reinforcement learning (RL) is now part of the state of the art in the domain of spoken dialogue systems (SDS) optimisation. Most performant RL methods, such as those based on Gaussian Processes, require to test small changes in the policy to assess them as improvements or degradations. This process is called on policy learning. Nevertheless, it can result in system behaviours that are not acceptable by users. Learning algorithms should ideally infer an optimal strategy by observing interactions generated by a non-optimal but acceptable strategy, that is learning off-policy. Such methods usually fail to scale up and are thus not suited for real-world systems. In this contribution, a sample-efficient, online and off-policy RL algorithm is proposed to learn an optimal policy. This algorithm is combined to a compact non-linear value function representation (namely a multilayers perceptron) enabling to handle large scale systems.

Mots clés: Spoken Dialogue Systems; Reinforcement Learning

Domaines: Apprentissage [cs.LG]; Machine Learning [stat.ML]

Lucie Daubigney, Matthieu Geist, Olivier Pietquin. Off-policy Learning in Large-scale POMDP-based Dialogue Systems. ICASSP 2012, Mar 2012, Kyoto, Japan. pp.4989-4992. ⟨hal-00684819⟩