<Phi2AI/>
Local AI Agents: 100% Offline with Gemma 3
Experiments

Local AI Agents: 100% Offline with Gemma 3

#Local-AI#Agents#Serbian#Gemma-3
System Architecture // Highlights
  • Offline First: The entire pipeline runs locally on consumer hardware (8GB VRAM) using Gemma 3.
  • Token Efficiency: Checking differences between Model tokenizers for Low Resource Languages.
  • Privacy Preserved: No data leaves the local network; best for critical and sensitive data.

Overview

The promise of AI Agents, autonomous systems that can plan, reason, and execute tasks is usually tied to massive cloud infrastructure and recently on powerful local machines. But for privacy-conscious research and low resource infrastucture, we expored systems that effectively fits in local deployment .

At MLC 25 Machine Learning Conference in Subotica, Serbia, (June ’25) we presented an experimental workflow for deploying Local AI Agents that can work on Serbian language. The goal: 100% offline capability on a standard laptop for Low-Resource-Languages.

The Infrastructure Constraint

Building local agents was a game of resource management, frameworks and engine. Unlike cloud API calls, every token costs VRAM. Our test environment was strictly limited to consumer specs:

  • GPU: NVIDIA RTX 4060 (8GB VRAM)
  • RAM: 32GB DDR5
  • Engine: Ollama (for model serving)

Model Selection: The Rise of Gemma 3

We tested three primary architectures for Serbian language support: Qwen, Mistral, and Gemma.

While Qwen-3-8B showed decent answers on Serbian, our research focused on Gemma 3 (4B). It fits perfectly within the 8GB VRAM buffer while leaving room for tool-calling and smooth run.

Fine-Tuning for Serbian

To improve the model’s native fluency, we fine-tuned Gemma 3 on a synthetic instruct dataset. The result is gemma-3-4b-it-sr, a model optimized for Serbian Q&A.

The Fine-Tuning Trade-off

While our fine-tuned model significantly improved Serbian fluency, we observed a degradation in “Agentic” capabilities, specifically in tool usage. This remains a key challenge in aligning specialized SLMs.

(HF Link - Gemma3-4B Serbian)

(HF - GGUF - Q8_0 (4.13G))

Implementation: The Agno Framework

To orchestrate these models, we utilized Agno (formerly Phidata), a lightweight agentic framework. Below is an implementation exapmle of a reasoning agent connected to a local SQLite database for memory.

Example:

from agno.agent import Agent
from agno.models.ollama import Ollama
from agno.storage import SqliteStorage

# Initialize the Serbian Reasoner
reasoning_agent = Agent(
    name="Reasoning Agent",
    model=Ollama(id="Sagicc/gemma-3-4b-it-sr"),
    reasoning=True,
    instructions=[
        "Always use tables to display data.",
        "Answer always on Serbian language."
    ],
    storage=SqliteStorage(
        table_name="reasoning_agent",
        db_file="agent_storage.db"
    ),
    markdown=True
)

Interactive Tool: Tokenizer Playground

Why do some models cost more to run than others? It often comes down to the Tokenizer. A tokenizer that isn’t optimized for Serbian will break words into many smaller fragments (tokens), increasing memory usage and latency.

We built a custom playground to test how different architectures tokenize Serbian text. Try any other non-English Language

Also available in the [Workbench] →

The Presentation

Explore the full workflow breakdown, including the “Task → Infrastructure → Agent” decision flow, in the slide deck below. Presented in MLC 25.

Artifact Resources

Back to Artifacts