Description
Title: LLM Routing: Selecting the Cheapest Capable LLM for Every Query
Abstract: Most LLM-based applications route every query to a single generalist model, typically a very large one, so that quality holds across all incoming queries. In practice, many of those queries are simple enough that a much cheaper model would answer them just as well. LLM routing reframes model choice as a per-query decision, sending each query to the cheapest model able to handle it. Routing can be framed as maximizing the quality of answers while minimizing inference costs. This raises challenges in real systems that we will review. Two of these challenges make router implementation difficult: the router must generalize to a rapidly evolving pool of LLMs, and it must not consume the resources it is meant to save. We focus on retrieval-based routers, a strong and low-resource baseline that meets both constraints: the routing decision reduces to a nearest-neighbour aggregation, and adapting to a new model pool means updating a corpus rather than retraining a router. We also present an efficient implementation in streaming settings.
Bio: AI Research Engineer @ Wikit & PhD CIFRE Student @ Laboratoire Hubert Curien, Université Jean Monnet