LLM Drift

The problem we're addressing is LLMs like ChatGPT changing unexpectedly with zero transparency. Current solutions like HuggingFace's OpenLLM leaderboard don't keep a history and require you to trust HuggingFace since they are the ones running the benchmarks and publishing the results. We solve this by using Galadriel which leverages teeML to trustlessly query ChatGPT and other AI models. We have written a generic framework for writing benchmarks based on Galadriel.

Based on this new framework, it's very easy to add new benchmarks and they will automatically be applied to all the LLMs being tracked. For example, I managed to write a new benchmark in just 5 minutes for counting happy numbers.

Mixture-of-expert models seem to be where the world is headed, where there are several models working together to achieve the best result. Different models are suited to different tasks, for example there may be a model that is better at mathematics. LLM Drift can be intelligent middleware that automatically routes user's requests to the best performing model based on their query. This can be integrated into LLM Routers like Openrouter (started by Alex Atallah).

Galadriel was instrumental to this project. It allows you to trustlessly access AI models via their TEE oracle from within smart contracts. The rest is simply solidity code to manage the different benchmarks and run them.

LLM Drift features abstractions that are suited to the specific problem of benchmarking LLMs for different tasks. For this, there is an IBenchmark interface to allow new benchmarks to be created and integrated very easily.

A node.js backend server serves a simple API that queries the Galadriel RPC for the latest runs of the LLMs and aggregate benchmark results.

Svelte is used on the frontend in combination with layercake for graphs.