A specialized 4B model improves ScallopBot tool selection

ScallopBot used a hosted model with more than 400 billion parameters for two tasks. The model selected tools and decided which conversation facts to keep.

We wanted to run those tasks locally with less delay. Over a weekend, we trained two versions of Qwen3.5-4B with LoRA, one for each task.

Training data and setup

The training data came from ScallopBot production traces. We removed identifying information and duplicate records. A larger teacher model supplied the labels.

The student models did not train on their own generated responses. We used rank 32, alpha 64, and two training epochs.

We evaluated the models on held-out turns. These turns were not part of the training data.

Tool selection results

scalloptools-1 selected the correct tool on 73.3 percent of the test turns.

Model	Correct tool selection
scalloptools-1	73.3 percent
Original 4B model	65.3 percent
Qwen3.6-35B	46.5 percent

The specialized model also performed better than the hosted model it replaced on this test. It was approximately 100 times smaller and four times faster.

All tool responses had a valid format. The model produced no fabricated tool calls in the test. It runs with 16 GB of RAM.

These results describe the evaluation set. They are not a guarantee for every future request.

Memory results

scallopmemory-1 reads a conversation and records facts that may be useful later. It can return no facts when a turn contains no useful memory.

Its teacher agreement score was 0.725, compared with 0.695 for the original model. It did not reach the teacher result of 0.748.

The smaller model responded in 4.2 seconds. The teacher took 31 seconds.

The result shows a tradeoff. Specialization improved the small model, but the teacher still performed better on this memory measure.

Two setup decisions

We disabled reasoning mode during inference. In this setup, explicit chain-of-thought output reduced performance.

We also limited the number of empty examples from each training session. Without that limit, the memory model learned to return no facts too often.

Try the models

Both models are available on Hugging Face under Apache 2.0. Each q5_k_m GGUF file is 3.16 GB.

They run in llama.cpp, Ollama, and LM Studio. Links and setup instructions are on the ScallopBot homepage.

This experiment suggests that small, specialized models can improve specific assistant tasks. The benefit depends on the task and the evaluation results.