Transducing Language Models

PhD defence by Vésteinn Snæbjarnarson

Assessment Committee

Professor Anders Søgaard, Computer Science, University of Copenhagen (Chairperson)
Ass. Professor David Chiang, University of Notre Dame, Dept. of Computer Science and Engineering
Professor Mirella Lapata, University of Edinburgh, School of Informatics and the NLP Processing Group

Supervisors

Professor Serge Belongie
Tenure-track assistant professor Ryan Cotterell

Department

Department of Computer Science

Place

The defence is conducted as a hybrid defence.

To attend the defence in person:
Building: Natural History Museum, Room: Main Auditorium,
Øster Voldgade 5 – 7, 1350 Copenhagen K

To attend the defence online:
Please follow the link to attend the defence online: https://ucph-ku.zoom.us/my/vesteinn
Password, if relevant: b1VxdFNoUytBa1NtZGorOWh5SnF3Zz09

Email address to gain access to the thesis: vest.snae@gmail.com.
You will either receive a copy of the thesis or be informed where you can read a physical copy.
Recipients of copies of the thesis are not allowed to share or distribute it due to copyright compliance.

Short description of the thesis

Language models define probability distributions over strings of symbols, but applications may need probabilities over different units. We might want to use a model trained on word fragments to derive probabilities for next bytes or whole words, or derive probabilities over protein sequences from a DNA model. This thesis addresses this mismatch by composing a pretrained model with a deterministic function encoded by a finite-state transducer, inducing a new language model without any parameter updates. The main challenge is to efficiently account for the probability mass of all sequences whose transformed outputs begin with a given target prefix. The thesis develops algorithms for exact inference when feasible, together with unbiased stochastic estimation of target-prefix probabilities when exact computation is impractical. The thesis combines formal language theory, probabilistic inference, and modern language models to show how pretrained models can be algorithmically adapted to downstream tasks.