Show HN: Any-LLM – Lightweight router to access any LLM Provider

AMeckes | 108 points

> LiteLLM: While popular, it reimplements provider interfaces rather than leveraging official SDKs, which can lead to compatibility issues and unexpected behavior modifications

with no vested interest in litellm, i'll challenge you on this one. what compatibility issues have come up? (i expect text to have the least, and probably voice etc have more but for text i've had no issues)

you -want- to reimplement interfaces because you have to normalize api's. in fact without looking at any-llm code deeply i quesiton how you do ANY router without reimplementing interfaces. that's basically the whole job of the router.

swyx | 12 hours ago

I'm excited to see this. Have been using LiteLLM but it's honestly a huge mess once you peek under the hood, and it's being developed very iteratively and not very carefully. For example. for several months recently (haven't checked in ~a month though), their Ollama structured outputs were completely botched and just straight up broken. Docs are a hot mess, etc.

amanda99 | 12 hours ago

How does this differ from this project? https://github.com/simonw/llm

pglevy | 10 hours ago

This looks awesome.

Why Python? Probably because most of the SDKs are python, but something that could be ported across languages without requiring an interpreter would have been really amazing.

piker | 12 hours ago

Crazy timing!

I shipped a similar abstraction for llms a bit over a week ago:

https://github.com/omarkamali/borgllm

pip install borgllm

I focused on making it Langchain compatible so you could drop it in as a replacement. And it offers virtual providers for automatic fallback when you reach rate limits and so on.

omneity | 10 hours ago

I have been also working on very similar open source project for python llm abstraction layer. I needed one for my research job. I inspired from that and created one for more generic usage.

Github: https://github.com/proxai/proxai

Website: https://proxai.co/

nexarithm | 11 hours ago

There is liteLLM, OpenRouter, Arch (although that’s an edge/service proxy for agents) and now this. We all need a new problem to solve

sparacha | 13 hours ago

I use Litellm Proxy, even in a dev environment via Docker, because the Usage and Logs feature greatly helps in providing visibility into LLM usage. The Caching functionality greatly helps in reducing costs for repetitive testing.

dlojudice | 12 hours ago

What is mozilla-ai?

Seems like reputation parasitism.

bdhcuidbebe | 8 hours ago

a proxy means you offload observability, filtering, caching rules, global rate limiters to a specialized piece of software - pushing this in application code means you _cannot_ do things centrally and it doesn't scale as more copies of your application code get deployed. You can bounce a single proxy server neatly vs. updating a fleet of your application server just to monkey patch some proxy functionality.

honorable_coder | 12 hours ago
[deleted]
| 8 hours ago

Anything like this, but in TypeScript?

klntsky | 11 hours ago

https://xkcd.com/927/ . LiteLLM rocks !

t_minus_100 | 11 hours ago

This is awesome, will give it a try tonight.

I’ve been looking for something a bit different though related to Ollama. I’d like a load balancing reverse proxy that supports queuing requests to multiple Ollama servers and sending requests only when a Ollama server is up and idle (not processing). Anything exist?

nodesocket | 11 hours ago

In truth it wasn’t that hard for me to ask Claude Code to just implement the text completion API so routing wasn’t that much of a problem.

renewiltord | 12 hours ago

Not to be confused with AnythingLLM.

weinzierl | 12 hours ago

Interesting timing. Projects like Any-LLM or LiteLLM solve backend routing well but still involve server-side code. I’ve been tackling this from a different angle with Airbolt [1], which completely abstracts backend setup. Curious how others see the trade-offs between routing-focused tools and fully hosted backends like this.

[1] https://github.com/Airbolt-AI/airbolt

mkw5053 | 12 hours ago