Fast, High-Fidelity LLM Decoding with Regex Constraints

Constraining the text generated by an LLM is essential for developing robust LLM applications. For example, a developer may require LLM responses to adhere to a specific JSON or YAML schema so that they are comprehensive and reliably parseable. Powerful tools and methods (Beurer-Kellner et al., 2022; Lundberg & Ribeiro, 2023; Willard & ... Read more 22 Feb 2024 - 16 minute read

An Optimal Lossy Variant of Speculative Decoding

Speculative decoding (Leviathan et al., 2023; Chen et al., 2023) is an elegant decoding strategy for autoregressive language models. It accelerates text generation while preserving the target distribution. In this blog post, I introduce mentored decoding, a novel, provably optimal, lossy variant of speculative decoding. It further increases the ... Read more 18 Sep 2023 - 8 minute read

Better Steering LLM Agents with LMQL

One of the most fascinating aspects of autoregressive large language models (LLMs) like GPT-3 is their ability to act through external tools. In this post, I’ll illustrate how LMQL (Beurer-Kellner et al., 2022), a new programming language for language model interaction, helps better steer such LLM agents. I’ll take ReAct (Yao et al., 2022) as an... Read more 28 Apr 2023 - 8 minute read

Adding Text (Really) inside Pictures and Videos

In the image above and the video below, the left character partially hides the text. In this post, I show how to automatically create such effect with a pretrained deep learning model and traditional computer vision techniques. I focus here on the general approach and you can see the implementation details (and create your own videos!) with the... Read more 08 Jun 2022 - 4 minute read

Simulating Depth Perception with Face Tracking

Motion parallax is created by the apparent relative motion of objects when an observer moves. It’s one of the monocular cues that enable depth perception. I created a demo to simulate this phenomenon thanks to TensorFlow.js and three.js. Just Show Me the Demo! Detecting the Orientation of the Camera/Face Axis Detecting the Distance betwe... Read more 04 May 2021 - 3 minute read

What Are the Ethical Risks of Your AI Project?

How to prevent AI systems from causing unintended harms? AI systems may cause unintended harms and their increasing adoption leads to additional risks and scrutiny. In this blog post, I present tools project teams can use to identify ethical concerns associated with their AI use cases. I first illustrate why this is so challenging, I then discu... Read more 20 Nov 2020 - 10 minute read

Learning through Auxiliary Tasks

Play, a seemingly futile activity, is ubiquitous in human cultures and very widespread in the animal kingdom, which suggests it serves an important purpose Update: this blog post was expanded into a paper with more explanations and additional experiments. In this post, I’d like to illustrate how auxiliary tasks can be beneficial for machine le... Read more 17 Feb 2019 - 19 minute read