Linux.com — Linux — Fri, 06 Jun 2025 10:53:28 +0000
Summary
This guide by Vedrana Vidulin of Intellias explains how to deploy lightweight language models on embedded Linux devices using LiteLLM, an open-source LLM gateway that acts as a proxy server providing a unified OpenAI-style API for local or remote models. The article walks through a five-step setup process: installing LiteLLM in a Python virtual environment on Debian, configuring models via a config.yaml file, serving models locally with Ollama (using the codegemma:2b model as an example), launching the LiteLLM proxy server, and testing the deployment with a Python script. It recommends compact models suited for resource-constrained hardware, including DistilBERT (66 million parameters), TinyBERT (14.5 million), MobileBERT (25 million), TinyLlama (1.1 billion), and MiniLM (33 million). Performance optimization tips include limiting response length with the max_tokens parameter, capping simultaneous requests with max_parallel_requests, implementing security measures like firewalls and authentication, and using LiteLLM's logging for monitoring. Running models locally on embedded devices offers reduced latency, improved data privacy, and offline functionality without relying on cloud services.
This article was contributed by Vedrana Vidulin, Head of Responsible AI Unit at Intellias (LinkedIn).
As AI becomes central to smart devices, embedded systems, and edge computing, the ability to run language models locally — without relying on the cloud — is essential. Whether it’s for reducing latency, improving data privacy, or enabling offline functionality, local AI inference opens up new opportunities across industries. LiteLLM offers a practical solution for bringing large language models to resource-constrained devices, bridging the gap between powerful AI tools and the limitations of embedded hardware.
Deploying LiteLLM, an open source LLM gateway, on embedded Linux unlocks the ability to run lightweight AI models in resource-constrained environments. Acting as a flexible proxy server, LiteLLM provides a unified API interface that accepts OpenAI-style requests — allowing you to interact with local or remote models using a consistent developer-friendly format. This guide walks you through everything from installation to performance tuning, helping you build a reliable, lightweight AI system on embedded Linux distribution.
Before you start, here’s what’s required:
First, we make sure the device is up to date and ready for installation. Then we install LiteLLM in a clean and safe environment.
Update the package lists to ensure access to the latest software versions:
| sudo apt-get update |
Check if pip (Python Package Installer) is installed:
| pip –version |
If not, install it using:
| sudo apt-get install python3-pip |
It is recommended to use a virtual environment. Check if venv is installed:
| dpkg -s python3-venv | grep “Status: install ok installed” |
If venv is intalled the output would be “Status: install ok installed”. If not installed:
| sudo apt install python3-venv -y |
Create and activate virtual environment:
| python3 -m venv litellm_envsource litellm_env/bin/activate |
Use pip to install LiteLLM along with its proxy server component:
| pip install ‘litellm[proxy]’ |
Use LiteLLM within this environment. To deactivate the virtual environment type deactivate.
With LiteLLM installed, the next step is to define how it should operate. This is done through a configuration file, which specifies the language models to be used and the endpoints through which they’ll be served.
Navigate to a suitable directory and create a configuration file named config.yaml:
| mkdir ~/litellm_configcd ~/litellm_confignano config.yaml |
In config.yaml specify the models you intend to use. For example, to configure LiteLLM to interface with a model served by Ollama:
| model_list: – model_name: codegemma litellm_params: model: ollama/codegemma:2b api_base: http://localhost:11434 |
This configuration maps the model name codegemma to the codegemma:2b model served by Ollama at http://localhost:11434.
To run your AI model locally, you’ll use a tool called Ollama. It’s designed specifically for hosting large language models (LLMs) directly on your device — without relying on cloud services.
To get started, install Ollama using the following command:
| curl -fsSL https://ollama.com/install.sh | sh |
This command downloads and runs the official installation script, which automatically starts the Ollama server.
Once installed, you’re ready to load the AI model you want to use. In this example, we’ll pull a compact model called codegemma:2b.
| ollama pull codegemma:2b |
After the model is downloaded, the Ollama server will begin listening for requests — ready to generate responses from your local setup.
With both the model and configuration ready, it’s time to start the LiteLLM proxy server — the component that makes your local AI model accessible to applications.
To launch the server, use the command below:
| litellm –config ~/litellm_config/config.yaml |
The proxy server will initialize and expose endpoints defined in your configuration, allowing applications to interact with the specified models through a consistent API.
Let’s confirm if everything works as expected. Write a simple Python script that sends a test request to the LiteLLM server and save it as test_script.py:
| import openai client= openai.OpenAI(api_key= “anything”, base_url= “http://localhost:4000 “)response= client.chat.completions.create( model= “codegemma”, messages= [{“role”: “user”, “content”: “Write me a Python function to calculate the nth Fibonacci number.”}])print(response) |
Finally, run the script using this command:
| python3 ./test_script.py |
If the setup is correct, you’ll receive a response from the local model — confirming that LiteLLM is up and running.
Optimize LiteLLM performance on embedded devices
Not every AI model is built for devices with limited resources — some are just too heavy. That’s why it’s crucial to go with compact, optimized models designed specifically for such environments:
Selecting a model that fits your setup isn’t just about saving space — it’s about ensuring smooth performance, fast responses, and efficient use of your device’s limited resources.
A few small adjustments can go a long way when you’re working with limited hardware. By fine-tuning key LiteLLM settings, you can boost performance and keep things running smoothly.
Restrict the number of tokens
Shorter responses mean faster results. Limiting the maximum number of tokens in response can reduce memory and computational load. In LiteLLM, this can be achieved by setting the max_tokens parameter when making API calls. For example:
| import openai client= openai.OpenAI(api_key= “anything”, base_url= “http://localhost:4000 “)response= client.chat.completions.create( model= “codegemma”, messages= [{“role”: “user”, “content”: “Write me a Python function to calculate the nth Fibonacci number.”}], max_tokens= 500 # Limits the response to 500 tokens)print(response) |
Adjusting max_tokens helps keep replies concise and reduces the load on your device.
Managing simultaneous requests
If too many requests hit the server at once, even the best-optimized model can get bogged down. That’s why LiteLLM includes an option to limit how many queries it processes at the same time. For instance, you can restrict LiteLLM to handle up to 5 concurrent requests by setting max_parallel_requests as follows:
| litellm –config ~/litellm_config/config.yaml –num_requests 5 |
This setting helps distribute the load evenly and ensures your device stays stable — even during periods of high demand.
A Few More Smart Moves
Secure your setup – implement appropriate security measures, such as firewalls and authentication mechanisms, to protect the server from unauthorized access.
Monitor performance – use LiteLLM’s logging capabilities to track usage, performance, and potential issues.
LiteLLM makes it possible to run language models locally, even on low-resource devices. By acting as a lightweight proxy with a unified API, it simplifies integration while reducing overhead. With the right setup and lightweight models, you can deploy responsive, efficient AI solutions on embedded systems — whether for a prototype or a production-ready solution.
Running LLMs on embedded devices doesn’t necessarily require heavy infrastructure or proprietary services. LiteLLM offers a streamlined, open-source solution for deploying language models with ease, flexibility, and performance — even on devices with limited resources. With the right model and configuration, you can power real-time AI features at the edge, supporting everything from smart assistants to secure local processing.
We’re continuously exploring the future of tech, innovation, and digital transformation at Intellias — and we invite you to be part of the journey.
Original article: https://www.linux.com/news/how-to-deploy-lightweight-language-models-on-embedded-linux-with-litellm/ Source: Linux.com