This post tells the story of how we compressed BGE-M3's Turkish capabilities by 880x — from 2,200 MB down to 2.5 MB — while retaining 92% of semantic accuracy and achieving 20,000+ sentences per second on CPU alone. No GPU required.

Along the way, we discovered something counterintuitive: quantizing the model to 2 bits resulted in no significant loss in performance. We'll explain why.

The Problem: Turkish NLP Deserves Better​

Turkish is an agglutinative language. A single root word like "göz" (eye) can produce hundreds of inflected surface forms: gözlerin, gözlerimde, gözlerininkini… Most multilingual models handle this by using massive shared vocabularies (BGE-M3 has 250,002 tokens) with subword tokenization. This works, but it comes with a 2+ GB memory footprint that excludes most real-world deployment scenarios.

We wanted a model that:

• Fits in a mobile app bundle (< 5 MB)
• Runs at 10,000+ sentences/second on CPU (no GPU)
• Retains > 90% of BGE-M3's semantic quality on Turkish STS benchmarks
• Requires zero external dependencies (no morphological analyzers, no Rust compilers)

Here's what we built:

As the benchmarks indicate, the 2-bit model maintains performance levels comparable to the FP16 version, showing no significant loss in accuracy.

Step 1: Distillation — Teaching a Lookup Table to Understand Turkish​

The Architecture​

Model2Vec is a technique that converts transformer sentence encoders into static embedding lookup tables. Instead of running 24 transformer layers per inference, you look up token embeddings from a matrix, average them, normalize, and you're done. O(1) per token, no attention heads, no GPU.

Our distillation pipeline works as follows:

Teacher: We used the pre-computed 1024-dimensional BGE-M3 sentence embeddings for all 533,189 Turkish Wikipedia articles from the dataset: hcsolakoglu/turkish_wikipedia_with_bge_m3_embeddings.

Dimensionality Reduction: We fit a 256-dimensional PCA on the teacher embeddings, preserving 86.77% of the total variance. This gives us compact, decorrelated teacher targets.

Student: A simple nn.Embedding(39655, 256) lookup table — just a matrix of shape 39,655 × 256, where 39,655 is the number of Turkish-relevant tokens we kept after pruning BGE-M3's multilingual vocabulary.

Training Objective: For each Wikipedia article, tokenize with the pruned vocabulary, compute weighted mean pooling over student embeddings, L2-normalize, and minimize cosine distance to the PCA-projected teacher representation.

Input Text → Tokenize → Lookup Embeddings → Weighted Mean Pool → L2 Normalize → Compare to Teacher

The Pruning Recovery Story​

This was the hardest part. When you prune a 250k multilingual vocabulary down to ~39k Turkish tokens, many subword units disappear. The tokenizer starts segmenting words differently, and previously one-token words become multi-token sequences. We call this subword segmentation drift.

The unpruned Model2Vec model (using the full 373k vocabulary with a morphological expander called Akana) scored 86.83% on STSb-TR at 182 MB. After pruning to 39k tokens, the naïve warm-start model crashed to 75.30% — an 11.5 point drop.

The key insight: don't freeze the embeddings; retrain everything from scratch. By training all 39,655 token embeddings end-to-end on 533k Wikipedia articles with gradient descent, the student model learns to compensate for segmentation drift. Each subword learns a representation that, when averaged with its new neighbors, produces the correct sentence-level embedding.

After 3 epochs of training, the pruned student model reached 91.36% — beating the 182 MB morphology-expanded model by +4.53 points at 9.4x smaller size.

Lesson learned: Explicit morphological expansion is unnecessary for Turkish Model2Vec. End-to-end distillation on real text heals the segmentation naturally.

Step 2: TurboQuant — Why Crushing Precision Improves Quality​

This is the surprising part. Google's TurboQuant (ICLR 2026) is a 3-stage vector quantization algorithm designed for KV-cache compression in LLMs. We adapted it for static sentence embeddings and discovered an unexpected benefit.

How TurboQuant Works​

Stage 1 — Random Orthogonal Rotation:

Before quantizing, TurboQuant multiplies the entire embedding matrix by a random orthogonal matrix R sampled from the Haar measure on O(d). This sounds like it should destroy information, but it actually improves the quantization properties.

Why? Embedding dimensions are typically anisotropic — a few dimensions carry most of the variance while most dimensions contain noise. The rotation redistributes variance equally across all dimensions, making each coordinate approximately Gaussian. This is critical for the next stage.

Stage 2 — Lloyd-Max Scalar Quantization:

Each rotated coordinate is independently quantized using optimal Lloyd-Max codebooks for the standard Gaussian distribution. For 4-bit quantization, this means 16 centroids per dimension; for 2-bit, just 4 centroids.

The Lloyd-Max quantizer minimizes mean squared error for Gaussian inputs. Because Stage 1 made all coordinates Gaussian, this is mathematically optimal — you get the minimum possible distortion for a given bit budget.

Stage 3 (Disabled) — QJL Residual Correction:

Google's original paper includes a 1-bit Quantized Johnson-Lindenstrauss sketch for the quantization residual. We found this hurts accuracy for sentence embeddings (1–2.5 point drop in Pearson r) because the JL error bound scales as O(1/√m), and at d=256, the sketch introduces more noise than it corrects.

The Denoising Effect: 2-bit Quantization Performance​

Here's the punchline. The Lloyd-Max quantizer helps maintain performance even at extremely low bitwidths:

• Small stochastic fluctuations in embedding coordinates (from gradient noise during training) sit near zero.
• The coarse 2-bit quantizer maps all near-zero values to the same centroid, effectively snapping noise to zero.
• Large, semantically meaningful values survive quantization because they fall into well-separated centroid bins.

The result: quantization preserves the angular structure (cosine similarity) that matters for semantic tasks while significantly reducing the model size. This robust architecture ensures that our 2-bit model (92.19%) performs on par with the unquantized FP16 model (91.36%), demonstrating that high-quality embeddings can be maintained even under extreme compression.
These results remain consistent across our evaluations on both the STSb-gorkem and STSb-emrecan datasets, demonstrating the robustness of our distilled model even under heavy compression.

Step 3: What Didn't Work (and What We Learned)​

Not everything was successful. Here's what we tried and discarded:

❌ Principal Component Removal (SIF)​

The classic unsupervised trick of subtracting the first principal component. This hurts our model because PCA teacher targets are already centered and orthogonalized — PC removal removes the principal semantic direction learned from BGE-M3.

❌ Orthogonal Procrustes Alignment​

Post-hoc rotation alignment between student and teacher spaces. Not needed because end-to-end training absorbs the optimal rotation directly into the embedding weights.

❌ QJL Residual Sketches​

As described above. Designed for MIPS in ultra-high dimensions (d=4096+), not for cosine similarity in d=256.

Try It Yourself​

All models are on Hugging Face and can be loaded in 2 lines of Python:

Load the FP16 Champion (19.36 MB)​

from model2vec import StaticModel

model = StaticModel.from_pretrained("altaidevorg/turkish-bge-m3-model2vec")

embeddings = model.encode([
    "Yapay zeka modelleri doğal dil işlemede çığır açıyor.",
    "İstanbul, Türkiye'nin en kalabalık şehridir."
])
print(embeddings.shape)  # (2, 256)

Load the 2-bit TurboQuant Model (2.50 MB)​

from src.turboquant import TurboQuantStaticModel

tq = TurboQuantStaticModel.from_pretrained(
    "altaidevorg/turkish-bge-m3-model2vec-turboquant-2bit"
)
embeddings = tq.encode(["Hafif modeller mobil cihazlarda harika çalışır."])
print(embeddings.shape)  # (1, 256)

Use 2-bit TurboQuant Model with Akana (2.50 MB)​

import akana

# TurboQuant 2-Bit Turkish Sentence Embeddings & Semantic Similarity
vec = akana.embed("Türkiye'nin başkenti Ankara'dır.")
print(f"Vector dim: {len(vec)}")  # -> 256

# Semantic cosine similarity
score = akana.similarity("ev", "evler")
print(f"Similarity: {score:.4f}")  # -> ~0.9130

# Batch embedding
vecs = akana.embed_batch(["Merhaba dünya", "Hava bugün çok güzel"])
print(f"Batch size: {len(vecs)}")  # -> 2

Use Akana from the CLI​

# Directly from cli
akana embed "Türkiye'nin başkenti Ankara'dır."
akana similarity "ev" "evler"

Load the 4-bit TurboQuant Model (4.92 MB)​

tq4 = TurboQuantStaticModel.from_pretrained(
    "altaidevorg/turkish-bge-m3-model2vec-turboquant-4bit"
)

Full Benchmark Results​

High-Precision Models​

Edge & On-Device Models (TurboQuant)​

Conclusion​

We compressed a 2.2 GB multilingual transformer into a 2.5 MB static embedding model that:

• Runs 800x faster on CPU
• Retains 92% of teacher quality on Turkish STS
• Requires zero GPU, zero morphological tools, zero external dependencies
• Fits inside a mobile app, a browser extension, or a smartwatch

The two key insights:

1. End-to-end distillation on real text beats morphological engineering. Don't enumerate surface forms; let gradient descent learn what subwords mean in context.
2. Quantization can be a regularizer, not just compression. TurboQuant's random rotation + coarse quantization acts as an implicit denoiser that removes training noise while preserving semantic structure.

Links​

• GitHub: github.com/altaidevorg/model2vec_experiments
• 🥇 FP16 Model (19.36 MB): altaidevorg/turkish-bge-m3-model2vec
• 🥈 4-bit Model (4.92 MB): altaidevorg/turkish-bge-m3-model2vec-turboquant-4bit
• 🥉 2-bit Model (2.50 MB): altaidevorg/turkish-bge-m3-model2vec-turboquant-2bit
• Training Dataset: hcsolakoglu/turkish_wikipedia_with_bge_m3_embeddings

Built by Altai. Licensed under Apache 2.0.

The examples exercise afterimage.simula, an experimental open implementation of Simula-style synthetic data mechanisms inspired by Davidson et al., Reasoning-Driven Synthetic Data Generation and Evaluation. They are not a Google reference implementation. The examples default to gemini-2.5-flash because it is fast enough for iterative taxonomy, scenario, and critic loops while staying close to the teacher-model family used in the paper.

There are three runnable scripts:

The Mental Model​

OpenSimula separates global coverage from local variety:

The generation loop in the minimal single-QA path looks like this:

instruction_y + optional documents
  -> build_taxonomy(y, S, D, N)
  -> infer_strategies(bundle)
  -> sample_mix(bundle, strategy)
  -> draw_meta_prompt(y, bundle, mix, K, complexify_c)
  -> generate_single_qa_datapoint(...)
  -> requirement critic accepts or refines
  -> DataPointRecord JSON

For MCQ, the final task step adds a second gate:

MCQ JSON
  -> requirement critic/refine loop
  -> double-critic probes the labeled answer
  -> accepted DataPointRecord or None

Imports: The Building Blocks​

The examples all start by creating an LLM provider:

from afterimage.providers import LLMFactory

llm = LLMFactory.create(
    provider="gemini",
    model_name="gemini-2.5-flash",
    api_key=api_key,
)

OpenSimula takes that provider and uses it for every structured LLM step: taxonomy proposals, critic merges, strategy inference, meta-prompts, task JSON, requirement critiques, refinements, and MCQ double-critic probes.

The smallest examples import:

from afterimage.providers import InMemoryDocumentProvider, LLMFactory
from afterimage.simula import OpenSimula, configure_example_console

You can read them as roles:

• LLMFactory creates the provider used by the OpenSimula facade.
• InMemoryDocumentProvider supplies the example policy excerpts as source material.
• OpenSimula orchestrates taxonomy, sampling, meta-prompt, and datapoint generation.
• configure_example_console() keeps progress output readable by muting noisy third-party INFO logs.

The batch script adds persistence helpers:

from afterimage.simula import (
    Checkpointer,
    OpenSimulaRunConfig,
    append_datapoints_jsonl,
    load_checkpoint,
)

These helpers are the difference between a toy single-row run and a reusable workflow. They let you save the expensive global scaffold once, resume from it, and append accepted task rows as generation completes.

Dataset Intent: instruction_y​

Each script defines one INSTRUCTION_Y string. This is the global dataset specification, usually called y in the paper mapping.

In minimal_pipeline.py, the intent is:

INSTRUCTION_Y = """\
You are generating synthetic **training Q&A** for enterprise employees...
"""

The wording does a lot of work:

• It names the audience: enterprise employees.
• It names the task format: training Q&A.
• It establishes grounding: answers must use the policy excerpts.
• It sets style and size constraints: factual tone, no panic language, word limits.
• It prevents unwanted hallucination: no vendor-specific products or laws that are not mentioned.

When adapting these examples, this is usually the first thing to rewrite. A good instruction_y is specific enough that an evaluator could reject rows that miss the mark.

Optional Documents: Source Grounding​

minimal_pipeline.py and corpus_batch_qa.py use static policy excerpts:

docs = InMemoryDocumentProvider(POLICY_EXCERPTS)

Those excerpts are used while building the taxonomy. They help OpenSimula infer factors and branches that are actually present in the domain material. In the minimal script, the source material is intentionally tiny. In the batch script, CORPUS_EXCERPTS has six policy-style snippets so the taxonomy has more surface area.

mcq_pipeline.py passes:

document_provider=None

That script demonstrates the no-corpus path. The taxonomy is inferred from instruction_y alone, which is useful when the target domain is a format or assessment style rather than a closed set of documents.

For real runs, replace InMemoryDocumentProvider with a provider that matches your source material. Directory, JSONL, or vector-backed providers are better fits once the corpus no longer fits comfortably in a Python list.

Console Setup and Progress​

The examples call:

configure_example_console()

OpenSimula makes many structured LLM calls. Without quiet logging, HTTP client and SDK messages can bury the useful progress signal. The examples also pass:

show_progress=True

to build_taxonomy(). That enables nested tqdm progress for factor proposal and per-factor breadth-first taxonomy expansion.

If a script appears paused after constructing OpenSimula, it is usually inside build_taxonomy(). The first structured call can take tens of seconds, and taxonomy expansion is sequential by design because each critic and planning step depends on the current tree state.

Taxonomy Construction​

All three scripts build a TaxonomyBundle:

bundle = await sim.build_taxonomy(
    INSTRUCTION_Y,
    document_provider=docs,
    target_depth_D=TARGET_DEPTH_D,
    proposal_N=PROPOSAL_N,
    max_factors=MAX_FACTORS,
    max_children_per_node=MAX_CHILDREN_PER_NODE,
    max_frontier_per_depth=MAX_FRONTIER_PER_DEPTH,
    show_progress=True,
)
OpenSimula.validate_taxonomy_bundle(bundle)

Important knobs:

The caps matter. A taxonomy is a branching structure; allowing too many factors, children, or frontier nodes can multiply into hundreds of sequential LLM calls. The examples choose conservative defaults so local runs are predictable.

The resulting bundle stores:

• the original instruction_y;
• document digests for bounded excerpts used during construction;
• accepted factors;
• one taxonomy tree per factor;
• expansion traces and per-depth plans for auditability.

Sampling Strategies and Mixes​

After the taxonomy is built, the scripts infer sampling strategies:

spec = await sim.infer_strategies(bundle)
mix = sim.sample_mix(bundle, spec)

The strategy spec describes which factors can be sampled together and with what weights. This prevents arbitrary combinations of taxonomy leaves from producing incoherent requirements.

sample_mix() draws one concrete Mix: a tuple of taxonomy nodes that becomes the requirement set for the next datapoint. In batch mode, every sample gets its own independent mix.

If you need full control, you can hand-author a SamplingStrategySpec, but the examples use infer_strategies() because it is the fastest way to get a reasonable mechanism from a new taxonomy.

Meta-Prompts: Local Diversity​

The sampled mix defines requirements. The meta-prompt turns those requirements into a local scenario:

meta = await sim.draw_meta_prompt(
    instruction_y=bundle.instruction_y,
    bundle=bundle,
    mix=mix,
    K=META_PROMPT_K,
    complexify_c=COMPLEXIFY_C,
    sequential=False,
)

K controls how many candidate scenarios are generated before one is subsampled. Larger values give the model more chances to vary framing while keeping the same global requirements.

complexify_c is the probability of running an additional complexification step. It is orthogonal to taxonomy coverage: complexity changes the difficulty or nuance of a scenario, while the mix controls which conceptual requirements are present.

The examples set sequential=False for speed. The sequential path generates scenario candidates one by one with prior attempts in context, which can reduce mode collapse at higher cost.

Single-QA Generation​

minimal_pipeline.py uses:

row = await sim.generate_single_qa_datapoint(
    instruction_y=bundle.instruction_y,
    bundle=bundle,
    mix=mix,
    meta=meta,
)

This produces one DataPointRecord with a single question and answer. The task generator first emits task JSON, then the requirement critic checks whether the row satisfies instruction_y, the sampled mix, and the meta-prompt.

If the critic rejects the row, OpenSimula can ask the LLM to refine it. The default maximum is four refine rounds. If the row still does not satisfy the requirements, the method returns None.

That is why the script handles both outcomes:

if row is None:
    print("No row accepted...")
else:
    print(row.model_dump_json(indent=2))

For a real dataset, treat None as a normal rejected sample, not an exception. Batch pipelines should count accepted rows and continue.

MCQ Generation and Double-Critic​

mcq_pipeline.py follows the same taxonomy, strategy, mix, and meta-prompt path. The task-specific call changes:

row = await sim.generate_mcq_datapoint(
    instruction_y=bundle.instruction_y,
    bundle=bundle,
    mix=mix,
    meta=meta,
    num_choices=NUM_CHOICES,
)

The MCQ path still runs the requirement critic and refinement loop. After that, it adds a double-critic gate for labeled-answer quality. The gate asks two independent structured probes about the answer label: one from the "this is correct" angle and one from the "this is incorrect" angle. A row is accepted only when the probes support a verifiable single correct answer.

Use this path when the output label matters as much as the question text. For free-form QA, the single-QA requirement critic is usually enough.

Batch Generation​

corpus_batch_qa.py turns the single-QA pattern into a multi-sample workflow:

async for _idx, rec in sim.aiter_single_qa_samples(
    instruction_y=bundle.instruction_y,
    bundle=bundle,
    spec=spec,
    n=args.num_samples,
    K=META_PROMPT_K,
    complexify_c=COMPLEXIFY_C,
    sequential=False,
    max_concurrency=args.max_concurrency,
    rng=rng,
):
    if rec is not None:
        append_datapoints_jsonl(jsonl_path, [rec])

Each sample independently draws:

mix -> meta-prompt -> task row -> critic/refine

max_concurrency bounds how many sample pipelines run at once. Increasing it can improve throughput, but each worker still performs several LLM calls. Start with 1 or 2 when using tight provider quotas.

Rows are appended as each async sample completes. This is deliberate: if a long run crashes after accepting some rows, those rows are already in data/train.jsonl.

Checkpoints and Resume​

The batch script saves the expensive global artifacts under opensimula/:

<output-dir>/
  opensimula/
    manifest.json
    taxonomy_bundle.json
    sampling_strategy.json
    run_config.json
  data/
    train.jsonl

The save path is:

with Checkpointer(out) as cp:
    bundle.save(cp)
    spec.save(cp)
    cp.write_run_config(run_cfg)

run_config.json is typed as OpenSimulaRunConfig; it records the model, temperature, taxonomy knobs, sample count, seed, output path, and corpus size. The manifest.json identifies the subtree as an AfterImage opensimula checkpoint.

On resume, the script skips taxonomy construction and strategy inference when possible:

ckpt = load_checkpoint(out)
bundle = ckpt.bundle
spec = ckpt.sampling_strategy

Use --resume when you want to append more rows using the same conceptual scaffold. This keeps follow-up generation cheaper and makes dataset expansion more consistent.

Running the Examples​

All scripts require GEMINI_API_KEY unless you edit the provider setup:

export GEMINI_API_KEY="your_api_key_here"

From examples/simula/:

python minimal_pipeline.py
python mcq_pipeline.py
python corpus_batch_qa.py --output-dir ./runs/corpus1 --num-samples 8 --max-concurrency 2
python corpus_batch_qa.py --output-dir ./runs/corpus1 --resume --num-samples 4

For Hub upload from the batch script, set HF_TOKEN or HUGGINGFACE_HUB_TOKEN, then pass:

python corpus_batch_qa.py --output-dir ./runs/corpus1 --push-hf username/dataset-name

The push path uploads the opensimula/ checkpoint subtree and a generated dataset README. It does not replace inspecting your local JSONL before scaling.

How To Adapt This Example​

Use this checklist when adapting examples/simula/ to another domain:

1. Rewrite INSTRUCTION_Y so it states the audience, task format, constraints, and non-goals.
2. Replace the static policy excerpts with realistic source material, or pass document_provider=None for an instruction-only taxonomy.
3. Start with target_depth_D=2, proposal_N=2 or 3, and capped factors while you inspect the taxonomy.
4. Validate the bundle after construction.
5. Infer strategies once per taxonomy bundle, then sample mixes many times.
6. Tune META_PROMPT_K and COMPLEXIFY_C only after the basic rows look grounded and on-format.
7. Use single-QA for free-form answers; use MCQ when labeled choices need the double-critic gate.
8. Save checkpoints for any run whose taxonomy cost you do not want to repeat.
9. Append accepted rows incrementally and treat rejected rows as expected sampling loss.
10. Inspect data/train.jsonl before increasing num_samples or concurrency.

Common Variations​

Use Files Instead of Inline Strings​

Replace InMemoryDocumentProvider with a provider that reads your corpus. Keep the rest of the OpenSimula flow the same:

documents -> taxonomy -> strategies -> mixes -> meta-prompts -> datapoints

Generate a Larger QA Dataset​

Use corpus_batch_qa.py as the starting point. Increase --num-samples, keep --max-concurrency conservative, and use --resume so you do not rebuild the taxonomy for every extension run.

Build a Multiple-Choice Benchmark​

Start from mcq_pipeline.py, strengthen INSTRUCTION_Y around answer verifiability, and keep num_choices=4 unless your downstream benchmark format requires a different shape.

Feed Simula Scenarios Into Multi-Turn Conversations​

OpenSimula can also prepare first-turn scenarios for ConversationGenerator via SimulaInstructionGeneratorCallback. That bridge does not call the LLM itself; it replays precomputed scenario text and metadata so the normal AfterImage conversation machinery can generate multi-turn assistant/user dialogs.

What These Examples Teach​

The reusable lesson is not "security policy Q&A." The reusable pattern is:

dataset intent
  -> optional source grounding
  -> factor taxonomies for global coverage
  -> weighted strategy mixes
  -> local meta-prompts for scenario diversity
  -> task-specific generation and critics
  -> checkpointed artifacts plus accepted JSONL rows

Once you understand that shape, you can apply OpenSimula to employee training, technical assessments, policy education, support simulations, benchmark-style MCQ generation, or any domain where you want explicit control over both coverage and local variation.

The script generates synthetic legal research conversations. For demonstration, we use a small slice of the free-law/Caselaw_Access_Project_embeddings dataset: cleaned U.S. court opinion text with precomputed BAAI/bge-base-en-v1.5 vectors. This keeps the example easy to run while still showing the full RAG data-generation pattern.

You can use the same structure with your own corpus. The important requirement is that index_corpus.py or your own ingestion job creates a Qdrant collection with document text in a known payload field and vectors that match the embedding model used later for query-time retrieval.

The Mental Model​

At a high level, the script wires together two related but different context flows:

That split is the main idea. The user side gets a sampled briefing so it can ask realistic questions. The assistant side gets retrieval results so it can answer from the corpus instead of inventing facts.

The generation loop then looks like this:

Qdrant collection
  -> QdrantDocumentProvider samples context
  -> ContextualInstructionGeneratorCallback creates user instructions
  -> QdrantRetriever retrieves answer context for each instruction
  -> WithRAGRespondentPromptModifier injects retrieved context
  -> ConversationGenerator simulates user/assistant turns
  -> JSONLStorage writes conversations

Imports: The Building Blocks​

The afterimage imports are the reusable pieces:

from afterimage import (
    ConversationGenerator,
    ContextualInstructionGeneratorCallback,
    EmbeddingProviderFactory,
    GenerationMonitor,
    WithRAGRespondentPromptModifier,
)
from afterimage.providers import QdrantDocumentProvider
from afterimage.retrievers import QdrantRetriever
from afterimage.storage import JSONLStorage

You can read them as roles:

• ConversationGenerator is the orchestrator-facing facade. It runs the simulated conversation and saves rows.
• ContextualInstructionGeneratorCallback creates the initial user questions from sampled document context.
• QdrantDocumentProvider is a document sampler. It gives AfterImage source text for instruction generation.
• QdrantRetriever is a semantic retriever. It searches Qdrant for passages relevant to the current instruction.
• WithRAGRespondentPromptModifier adds retrieved passages to the assistant prompt before the assistant answers.
• EmbeddingProviderFactory creates the query embedding backend used by QdrantRetriever.
• GenerationMonitor records generation metrics and alerts.
• JSONLStorage writes the final dataset.

The Qdrant client imports are infrastructure rather than AfterImage concepts:

from qdrant_client import AsyncQdrantClient, QdrantClient

This example uses the sync client for document sampling and the async client for retrieval.

Prompts: Defining the Two Actors​

The script defines two system prompts:

RESPONDENT_PROMPT = """You are a careful senior legal research assistant..."""
CORRESPONDENT_PROMPT = """You are an experienced lawyer or a legally curious client..."""

In AfterImage terms:

• The respondent is the assistant being trained or simulated.
• The correspondent is the user side of the conversation.

The respondent prompt is strict about grounding:

• Use retrieved court opinions and excerpts.
• Preserve identifiers when available.
• Explain in plain English.
• Say what is missing when the context is insufficient.
• Do not provide real-world legal advice.

The correspondent prompt shapes the simulated user:

• Ask realistic legal questions.
• Stay in role.
• Do not invent docket numbers or citations.
• Use the provided briefing as inspiration.

When adapting this example, these prompts are usually the first thing to change. For customer support, the respondent might be a support agent and the correspondent might be a customer. For medical education, the respondent might be a study tutor and the correspondent might be a student. The architecture can stay the same while the roles change.

CLI Arguments: Making the Example Reusable​

_build_parser() exposes the parts you are likely to tune:

p.add_argument("--qdrant-url", ...)
p.add_argument("--collection", ...)
p.add_argument("--num-dialogs", ...)
p.add_argument("--max-turns", ...)
p.add_argument("--max-concurrency", ...)
p.add_argument("--gemini-model", ...)
p.add_argument("--embedding-model", ...)
p.add_argument("--output", ...)

Most flags also read from environment variables. That makes the same script usable in local demos, scheduled jobs, and documentation examples without editing source code.

Important knobs:

Qdrant Connection​

The helper _qdrant_kwargs() builds shared client settings:

def _qdrant_kwargs(url: str, api_key: str | None) -> dict:
    kwargs: dict = {"url": url, "timeout": 120.0}
    if api_key:
        kwargs["api_key"] = api_key
    return kwargs

Both local Qdrant and Qdrant Cloud use the same shape. Cloud adds an API key; local Docker usually does not.

In _async_main(), the script creates both clients:

qd = QdrantClient(**qd_kw)
qd_async = AsyncQdrantClient(**qd_kw)

The async client is closed in finally:

await qd_async.close()

That pattern is worth copying for any script that owns network clients.

Monitoring and Alerts​

The monitor records metrics such as generation time, success rate, error rate, token usage, and conversation length:

monitor = GenerationMonitor(
    log_dir=str(args.log_dir),
    alert_handlers=[on_alert],
    metrics_interval=60,
)

In this example, alerts are printed:

def on_alert(alert) -> None:
    print(f"alert - {alert.name} - {alert.message}")

For production workflows, this could send alerts to a dashboard, Slack, logs, or your own observability pipeline. For documentation and demos, printing is enough because it shows when generation quality or provider reliability is degrading.

Instruction-Side Context​

This block creates the document provider:

documents = QdrantDocumentProvider(
    client=qd,
    collection_name=args.collection,
    content_key=args.content_key,
    max_docs=args.max_docs,
)

QdrantDocumentProvider gives AfterImage documents to sample from. It is not the same as retrieval. It is used before the conversation starts, so the instruction generator can create grounded user questions.

Then the script creates the instruction callback:

instruction_cb = ContextualInstructionGeneratorCallback(
    api_key=api_key,
    documents=documents,
    model_name=args.gemini_model,
    num_random_contexts=1,
    llm_create_extras={...},
)

This callback asks the LLM to produce user instructions from sampled context. Its output is a GeneratedInstructions object containing:

• instructions: one or more user prompts.
• context: the sampled source text.
• context_id / context_ids: metadata for coverage tracking.

In your own project, swap QdrantDocumentProvider for another provider if your source material lives somewhere else:

• InMemoryDocumentProvider for small examples.
• DirectoryDocumentProvider for local files.
• JSONLDocumentProvider for prepared document rows.
• QdrantDocumentProvider for vector database-backed corpora.

Query Embeddings​

The retriever needs to embed user instructions before searching Qdrant:

embedding_provider = EmbeddingProviderFactory.create(
    {
        "type": "process",
        "model": args.embedding_model,
        "workers": args.embedding_workers,
    },
)

This example uses a local SentenceTransformer process pool. That is why setup requires:

uv sync --extra embeddings-local

The important rule is that query embeddings must be compatible with indexed vectors. Here, index_corpus.py stores 768-dimensional BAAI/bge-base-en-v1.5 vectors, so generate.py defaults to the same model.

If you index with a different model, change both indexing and querying together. Otherwise Qdrant search may fail or return poor matches.

Response-Side Retrieval​

The retriever is built separately from the document provider:

retriever = QdrantRetriever(
    client=qd,
    collection_name=args.collection,
    embedding_provider=embedding_provider,
    async_client=qd_async,
    payload_key=args.content_key,
    limit=3,
)

This retrieves the top matching excerpts for a generated instruction. limit=3 means the assistant will see up to three retrieved passages.

Then the retriever is wrapped in a respondent prompt modifier:

modifier = WithRAGRespondentPromptModifier(retriever=retriever)

The modifier runs before the assistant answers. It takes the base respondent prompt and adds retrieved context, so the assistant response is grounded in search results.

This is the pattern to copy when you want RAG-style synthetic conversations:

DocumentProvider -> helps produce the user question
Retriever        -> helps produce the assistant answer
PromptModifier   -> injects retrieved answer context

Storage​

The generated rows are written to JSONL:

storage = JSONLStorage(conversations_path=str(args.output))

JSONL is a good default because it is easy to inspect, stream, and convert later. Each row contains the conversation plus metadata such as sampled context, retrieved context, and context ids.

If your workflow needs a database, AfterImage also supports SQL storage. The rest of the generation composition can stay mostly the same.

The ConversationGenerator​

This is where the pieces come together:

conv_gen = ConversationGenerator(
    respondent_prompt=RESPONDENT_PROMPT,
    correspondent_prompt=CORRESPONDENT_PROMPT,
    api_key=api_key,
    model_name=args.gemini_model,
    monitor=monitor,
    auto_improve=args.auto_improve,
    storage=storage,
    instruction_generator_callback=instruction_cb,
    respondent_prompt_modifier=modifier,
    embedding_provider=embedding_provider if args.auto_improve else None,
    llm_factory_kwargs={...},
)

The most important fields are:

This construction is the reusable recipe. The caselaw dataset is just one instance of it.

Running Generation​

The actual generation call is short:

await conv_gen.generate(
    num_dialogs=args.num_dialogs,
    max_turns=args.max_turns,
    max_concurrency=args.max_concurrency,
)

num_dialogs controls how many conversations to produce.

max_turns is not "always exactly this many turns." AfterImage samples the actual turn count uniformly from 1 through max_turns. With max_turns=1, every conversation is single-turn.

max_concurrency controls how many conversation workers run at once. Higher values can improve throughput but also increase API pressure. For Gemini free or tight quotas, 1 is the safest starting point.

Cleanup​

The script closes resources in a finally block:

finally:
    await embedding_provider.aclose()
    await qd_async.close()
    monitor.shutdown()

This matters because generation scripts often run for a long time and own process pools, async HTTP sessions, and monitoring threads.

If you add another resource, such as a database client or custom retriever, close it here too.

How To Adapt This Example​

Use this checklist when adapting the example to another domain:

1. Replace RESPONDENT_PROMPT with the assistant behavior you want.
2. Replace CORRESPONDENT_PROMPT with the user role you want to simulate.
3. Choose a document provider for your source material.
4. Make sure your indexed vectors and query embedding model match.
5. Choose a retriever and prompt modifier if the assistant should use RAG.
6. Pick storage: JSONL for files, SQL for database-backed runs.
7. Start with low num_dialogs and max_concurrency.
8. Turn on auto_improve only when you are ready to pay for quality retries.
9. Inspect output JSONL before scaling up.

Common Variations​

Use local files instead of Qdrant for instruction context​

If you do not need vector-backed sampling, use a file or directory provider for the instruction side. You can still use a retriever for response-side RAG, or skip retrieval entirely and use WithContextRespondentPromptModifier for simpler context injection.

Generate customer support data​

Keep the same shape:

support articles -> document provider -> customer questions
support articles -> retriever -> agent answers

Change the prompts so the respondent is a support agent and the correspondent is a customer with realistic issues.

Generate domain tutoring dialogs​

Use textbook sections, lecture notes, or documentation pages as source material. The correspondent can be a beginner, advanced learner, or examiner. The respondent can be a tutor that explains concepts and asks clarifying questions.

Skip RAG for simpler datasets​

If every generated answer can rely on the same sampled context, you can remove QdrantRetriever and WithRAGRespondentPromptModifier, then use a simpler context prompt modifier. The tradeoff is less dynamic answer-time retrieval.

What This Example Teaches​

The important lesson is not "caselaw plus Qdrant." The reusable pattern is:

source material
  -> instruction generation
  -> optional retrieval
  -> controlled assistant/user simulation
  -> monitored JSONL dataset

Once you understand those pieces, you can assemble your own synthetic dataset pipeline for legal research, customer support, technical documentation, medical education, internal knowledge bases, or any domain where conversations should be grounded in source documents.

Originally published on Medium

Today we are releasing AfterImage as open source software under the Apache 2.0 license, with packages on PyPI (pip install afterimage / uv add afterimage) and documentation at afterimage.altai.dev.

If you have ever tried to scale instruction tuning or evaluation data with an LLM, you have probably felt the same tension: raw volume is easy; useful volume is not. AfterImage exists to treat synthetic dataset generation as a systems problem — one where you can steer grounding, diversity, structure, and quality instead of hoping a clever prompt loop will "just work."

Why we built it​

Modern LLM workflows lean heavily on synthetic data: bootstrapped instructions, evolved difficulty, self-curation, RAG-style corpora, and more. The research and practice around these methods make a consistent point easy to miss in day-to-day engineering: the value of synthetic data is not automatic. Without controls, you get shallow diversity, weak grounding to sources, evaluator leakage, and answers that sound right but are not anchored in anything you trust.

When we first designed the architecture of AfterImage internally, the core idea was simple to state and hard to operationalize: synthetic data quality has to be measured and steered, not assumed. The system was designed around explicit roles, composable stages, schema constraints where you need them, and monitoring that does not get in the way of throughput.

That design is still the spine of the project. What changed since that early write-up is everything around it: a YAML-driven CLI for getting JSONL in one command, export to common fine-tuning shapes, preference / DPO-style pair generation, broader provider support (including local OpenAI-compatible servers), a public docs site, and a codebase that has been exercised on real pipelines rather than described only on paper.

What AfterImage actually does​

AfterImage simulates conversations between two modeled roles:

• Correspondent — the side that initiates: questions, tasks, follow-ups. Behavior is driven by instruction generators and can be shaped by personas (tone, expertise, intent) so your dataset is not "one generic user, forever."
• Respondent — the assistant side, defined by your system prompt and runtime prompt modifiers (for example, injecting retrieved chunks for RAG-style grounding).

Generation is async-first, so you can run concurrent workers and make better use of API throughput. Persistence (JSONL or SQL) is decoupled from the core loop, so the same generation logic can land in different environments without rewrites.

There are two "front doors," intentionally:

1. CLI + config — describe a run in YAML, set keys in the environment, run afterimage generate. Dry-run with --dry-run when you want to validate the plan without spending tokens. When you are ready to train, afterimage export converts datasets into formats like ShareGPT-style or Hugging Face messages layouts; afterimage preference helps produce preference pairs for alignment workflows.

2. Python API — compose instruction generators, respondent prompt modifiers, stopping criteria, storage, judges, and monitoring the same way the CLI does under the hood. The API is where specialized flows live: document-grounded instruction, structured extraction, tool-calling-oriented setups, custom storage, and tighter integration with your stack.

If you want a quick way that your coding agents can get up and running with AfterImage, the project publishes llms.txt with install steps, CLI entry points, and links to the Markdown guides and examples.

Grounding, structure, and quality loops​

Document providers can feed local files, JSONL, in-memory lists, or Qdrant — so "what the model is allowed to know" can be a first-class input to generation, not an afterthought pasted into a prompt.

Structured generation uses Pydantic schemas so single-turn outputs are valid JSON-shaped objects: extraction from unstructured text, synthetic rows with typed fields, or evaluation artifacts you can consume downstream without brittle parsing.

Evaluation is meant to be practical at scale: embedding-based signals for fast filtering (coherence, relevance, grounding-style checks where configured) and LLM-as-judge style rubrics when you need a second pass. The framework also supports quality gates and regeneration paths (for example, auto_improve workflows) when you want the generator to retry or revise under explicit criteria rather than silently accepting bad rows.

Monitoring tracks latency, token usage, errors, and evaluation signals over time so long runs remain operable — synthetic data generation is as much an engineering workload as a modeling one.

Design choices that survived contact with reality​

Two ideas from the original technical framing stayed true in the shipped library:

Composition over inheritance. You should not need to fork a monolithic "generator" class for every new corpus or policy. Behavior is injected through callbacks and strategies — instruction logic, prompt modification, storage, evaluation — so the core stays stable while your pipeline evolves.

Provider portability. AfterImage normalizes chat sessions, structured calls, token accounting, and metadata across Gemini, OpenAI-compatible APIs (including DeepSeek and OpenRouter where applicable), and local OpenAI-compatible servers (for example vLLM, Ollama, llama.cpp). Conversation state is managed independently of any single vendor's chat API quirks.

Internally, recent work strengthened the parts that matter most in production: smarter context and persona sampling (including coverage-oriented behavior), clearer separation between orchestration, sampling, and quality gating, and optional capture of reasoning / thinking content from compatible providers when you want that signal preserved in your dataset.

What AfterImage is not​

AfterImage is not a promise that synthetic data is "as good as hand-labeled gold for all sorts of use cases" without validation, and it is not a theory of instruction learning. It is a pragmatic engine for people who need repeatable, configurable, observable generation — from a quick experiment to a large batch job with budgets and filters.

The same limitations that applied when we wrote the internal report still apply in public form: evaluator bias can leak into curated sets; LLM judges cost time and money; personas need care so diversity does not collapse into caricature. We ship controls and visibility so you can see those trade-offs instead of discovering them only after training.

Try it​

• Repository: github.com/altaidevorg/afterimage
• Documentation: afterimage.altai.dev
• Install: Python 3.11+, then pip install afterimage or uv add afterimage
• Quick start: afterimage generate -c examples/configs/basic.yaml (see examples/configs/ for RAG, local models, stopping budgets, and more)

Optional extras cover local embeddings (embeddings-local), a small FastAPI server (server), and a training / demo stack (training) for the Gradio demo and training helpers — see pyproject.toml for the exact dependency sets.

Closing​

Open sourcing AfterImage is our invitation to treat synthetic data as infrastructure: something you configure, measure, and iterate on like any other part of the ML stack. If you build something with it — or hit a wall we should address — we would love to hear from you in issues and discussions on GitHub.

Originally published on Medium

Open-source language models have made incredible progress in reasoning and instruction following — yet they still struggle with one crucial skill: evaluation. How can we train small models to judge long-form text with the nuance and consistency of giant models such as GPT-4?

In this project, we built a compact, open-source LLM-as-a-Judge by fine-tuning the Gemma 3 4B model on two complementary datasets from Prometheus-Eval:

• Feedback Collection → absolute scoring (1–5)
• Preference Collection → pairwise ranking (A vs B)

We trained two specialized models, then merged them with Nuslerp using the MergeKit library — creating a single, balanced evaluator capable of both scoring and ranking. With extensive evaluation and testing of this model, we found out that it outperforms the previously open-source SOTA model fine-tuned on this dataset despite being much smaller. We also measured the consistency among its evaluations and verified that it does a very good job without a specialized heuristic method.

Why Build an LLM-as-a-Judge?​

Human evaluation remains the gold standard for measuring helpfulness, factuality, and coherence in model outputs. However, it is expensive, slow, and difficult to scale. We need automated evaluations to be able to increase the number of iteration cycles and make rapid progress.

LLM-judges fill this gap by learning to approximate human preferences through structured feedback data. A good judge model can:

• Score model outputs on dimensions like helpfulness, factuality, etc.
• Compare two responses and decide which one is better.
• Provide textual rationale for its judgment.
• Serve as an automated evaluation system for ongoing model training experiments.

Datasets: Prometheus-Eval Feedback & Preference Collections​

Both datasets were released in Prometheus (2024) to teach open models fine-grained evaluation capabilities.

Feedback Collection — Absolute Scoring​

• 1K rubrics, 20K instructions + reference answers, 100K responses + feedback
• Output format — Feedback: <critique text> [RESULT] <1-5>
• Trains models to assess responses against reference answers and rubric descriptions.

Preference Collection — Pairwise Judgment​

• 1K rubrics, 20K instructions, 200K response pairs (A/B)
• Output format — Feedback: <comparison text> [RESULT] <A or B>
• Trains models to compare two responses and pick the superior one.

Dual Fine-Tuning Strategy​

We fine-tuned two separate judge models:

• Feedback model on Feedback Collection
• Preference model on Preference Collection

We fine-tuned attention and MLP modules of the first model using LoRA with r = 64, LoRA alpha = 32, and LoRA dropout = 0.15 on top of the feedback collection. In parallel, we fine-tuned another model on top of the preference collection with R = 128.

Each model captured distinct judgment behavior: one mastered calibrated scoring, the other comparative reasoning.

Merging with Nuslerp (MergeKit)​

To combine their complementary abilities, we used Nuslerp, a non-linear, spherical interpolation algorithm that merges weight spaces layer-wise for smoother semantic blending.

mergekit --base unsloth/gemma3-4b-it \
         --models feedback_model preference_model \
         --weights 0.5 0.5 \
         --method nuslerp \
         --output gemma3-4b-judge-merged

Evaluation Results​

We evaluated all models on Prometheus-Eval/Feedback-Bench and Prometheus-Eval/Preference-Bench.

Feedback Model Results​

Preference Model Results​

Merged Model (Feedback + Preference)​

Base Model (google/gemma3-4b-it)​

Comparison Summary​

Our merged Gemma 3 4B-based judge matches or surpasses Prometheus 2 (8×7B) correlations while being significantly smaller.

Consistency Evaluation​

TrustJudge is a probabilistic evaluation framework designed to make LLM-as-a-judge systems more reliable. Traditional evaluation setups often suffer from two key issues:

1. Score-Comparison Inconsistency — where lower-rated responses can outperform higher-scored ones in pairwise comparisons.
2. Pairwise Transitivity Inconsistency — where circular or contradictory preferences emerge (e.g., A > B > C > A).

To address these, TrustJudge introduces distribution-sensitive scoring, which converts discrete rating probabilities into continuous expectations for finer-grained judgments, and likelihood-aware aggregation, which corrects transitivity violations using bidirectional preference probabilities.

We experimented with the TrustJudge approach in our case. Although the method theoretically enhances evaluation consistency, in our experiments it didn't lead to any noticeable difference. This suggests that TrustJudge, despite being beneficial for zero-shot settings, may not provide additional benefits for task-specific fine-tuned models, since such models have already internalized the key evaluation signals during training.

Inconsistency in TrustJudge​

We used the prometheus-eval/Preference-Bench dataset to analyze evaluation inconsistencies. For each sample in the dataset, we evaluated two responses and selected the one with the higher score. Then, we checked whether this response had been previously preferred over the other one. If not, we marked it as an inconsistent sample.

After running this process across the entire dataset:

Limitations & Next Steps​

Even though the model also does a decent job in non-English settings thanks to Gemma 3's multilingual capabilities, we did not run systematic benchmarks in these languages since the underlying dataset is exclusively in English. A natural next step will be to fine-tune the model explicitly for multilingual use cases.

The current models need reference answers to evaluate model responses. This is particularly useful to evaluate models against a validation subset where we already have a golden answer. However, another common use case is to evaluate responses supposed to be grounded on a given reference context. We will add support for such use cases as an upcoming improvement.

Links​

• Gemma Judge on Hugging Face
• Altai Discord Server

Originally published on Medium

At Altai, we're frequently asked one question:

"Why bother fine-tuning when you can just use RAG?"

Retrieval-Augmented Generation (RAG) — the idea of feeding external documents to a frozen large language model (LLM) — quickly became the default pattern for domain-specific enterprise AI. It's conversational, so it sounds intuitive to even non-technicals. It avoids retraining, so it's developer-friendly. It's flexible and composable, so you can build different pipelines with multiple components.

But something has been missing in that story: it's simply like giving a complex textbook to someone without the background and expecting expert reasoning — they can find surface-level answers to trivial questions, but they will struggle to synthesize deeper, domain-grounded insights and inevitably fall short on higher-order reasoning.

A recent academic paper has empirically shown this to be true. The authors of The Role of Parametric Injection — A Systematic Study of Parametric Retrieval systematically compared:

• Contextual RAG — standard RAG with retrieval results injected in the context
• PRAG — knowledge injected in the form of LoRA adapters trained on documents
• PRAG-Combine — a hybrid method combining both

With this study, they concluded that RAG gets smarter, more robust, and more faithful when combined with domain-specific fine-tuning, which supports the hypothesis we built Altai around.

The core insight: synthetic QA fine-tuning improves RAG itself​

The authors fine-tuned LoRA adapters (small, efficient fine-tuning modules) on document-question-answer triples — many of which were synthetically generated. These adapters captured domain-specific semantics, effectively encoding how to interpret retrieved text, not just what the text says.

Then they tested three setups:

• RAG: Retrieve relevant text and prompt the base model. Baseline.
• PRAG: Inject fine-tuned LoRAs, no retrieval. Worse than RAG — lacks fine-grained factual details.
• PRAG-Combine: Use both fine-tuned LoRAs and retrieval. Outperformed RAG on all benchmarks. Excels in the case of noisy retrieval results.

The hybrid model — PRAG-Combine — consistently beat RAG across factual QA, multi-hop reasoning, and robustness tests with noisy or irrelevant passages.

That's the headline:

"Fine-tuning with synthetic QA pairs doesn't compete with RAG — it amplifies and completes it exactly where it fails."

Why this matters for Altai​

At Altai, we've built a platform where companies can train small LLMs on synthetic, domain-specific QA datasets — generated automatically from their own knowledge base.

The intuition has always been simple:

"You can't expect a model that's never seen your domain to interpret your documents optimally — even if you retrieve the right ones."

This new research provides empirical backing for that intuition. It shows that domain fine-tuning changes how the model uses retrieved context. LoRAs trained on QA pairs help the model integrate text more coherently, reduce hallucination, and stay grounded even when retrieval is messy.

Altai's approach — generating synthetic QAs, fine-tuning, and combining that with retrieval — aligns exactly with PRAG-Combine, the configuration that outperformed everything else.

What's actually happening under the hood​

The paper's layer-wise analysis revealed the mechanism that explains why PRAG-Combine works better: Fine-tuned adapters increase parametric knowledge scores in the later layers of the transformer — the parts responsible for semantic reasoning.

In plain terms, fine-tuning on synthetic QA pairs teaches the model "how to think" about domain-specific information.

That means when the retriever feeds a new document, the model doesn't just read it literally — it interprets it through the lens of its domain knowledge. It knows what really matters, resolves conflicts, and synthesizes a thoughtful response instead of copy-pasting from the context.

In the Altai pipeline, this is exactly what happens:

1. Synthetic QA generation: Altai's Afterimage framework leveraging our custom-made models generates diverse, domain-relevant questions and answers from a company's documents.
2. Fine-tuning adapters: These QA pairs train a smaller model or LoRA module to internalize the domain's structure — concepts, relationships, typical instruction forms.
3. Hybrid inference (letsearch): During serving, we still use retrieval — but now the model interprets the retrieved chunks with domain intuition.

Result: faster convergence, better grounding, and much more reliable outputs in specialized domains (law, healthcare, finance, etc.).

Why synthetic QA data works so well​

Synthetic QA pairs are not just convenient — they're structurally ideal training data:

• Question form teaches information need recognition — what matters in the text.
• Answer form teaches fact grounding — where and how to extract it.
• Together, they encode "retrieval intent" directly into the model's parameters.

The paper shows that even when generated automatically, such QA data improves interpretive alignment between the model and retrieved context.

In other words, the model learns not just facts, but also how to think in the light of facts.

The big picture: fine-tuning and retrieval aren't rivals​

The industry treated RAG as the "fine-tuning killer." However, this paper demonstrates that RAG does not suffice in settings involving domain knowledge. The solution isn't RAG or fine-tuning — it's RAG plus domain fine-tuning via synthetic data.

Altai operationalizes that theoretical finding. Our platform:

• Generates synthetic QA datasets tailored to your documents thanks to our enterprise-grade synthetic dataset engine (Afterimage).
• Fine-tunes or LoRA-adapts models to embed domain semantics on the Altai platform.
• Combines that with retrieval through our RAG layer (letsearch) for deployment.

The result: leaner models that know your domain and interpret your knowledge base intelligently. Retrieval gets you access to knowledge, but fine-tuning teaches you how to use it.

Altai's synthetic QA fine-tuning turns your documents into domain intuition — the missing half of the retrieval story.

Originally published on Medium

In today's rapidly evolving digital landscape, businesses of all sizes are eager to harness the transformative power of Artificial Intelligence (AI). Large Language Models (LLMs) like GPT-4 or Gemini have captured headlines with their broad capabilities, and for good reason. However, the initial fascination with these massive, general-purpose models is giving way to a more nuanced understanding of their practical limitations for many enterprise applications. This is where Altai's Small Language Models (SLMs) emerge as a game-changer, offering a pragmatic and powerful alternative designed specifically for business needs.

Even tech giants like NVIDIA acknowledge this shift. As a recent research paper from NVIDIA emphasizes, "SLMs are sufficiently powerful, inherently more suitable, and necessarily more economical for many invocations in agentic systems," suggesting that small models — not large — will be the backbone of scalable and sustainable AI systems in the future (Belcak et al., 2025).

The Problem with "Bigger is Better": Why LLMs Fall Short for Businesses​

While LLMs are remarkable in their versatility and extensive linguistic knowledge, their inherent design often presents significant drawbacks for corporate adoption:

• Prohibitive Costs: Training and operating LLMs are incredibly expensive due to their immense scale and computational demands. They require high-end GPUs, large infrastructure, and consume substantial energy, leading to high cloud computing bills and significant ongoing operational expenses. For instance, training a model like GPT-3 could cost approximately $1.4 million per session, and per-query costs can be as high as $0.09 compared to less than $0.0004 for SLMs.

• Inefficiency for Specific Tasks: LLMs are generalists, designed to emulate broad human intelligence across diverse domains. While versatile, this broadness can paradoxically make them less precise or even lead to "hallucinations" — generating factually incorrect or irrelevant information — when applied to highly specialized fields without specific adaptation. This makes them less reliable for critical business functions like legal analysis or medical diagnostics where precision is paramount.

• Significant Data Security Concerns: Most leading LLMs (like GPT, Claude, Gemini) are cloud-based and process user data through third-party providers. This poses substantial data security and privacy risks for businesses handling sensitive or regulated information, as data leaves the company's control. Compliance with regulations like GDPR or HIPAA becomes a complex challenge.

Altai's "Right-Sized AI" Solution: The Power of Domain-Specific SLMs​

Altai directly addresses these challenges by focusing on Small Language Models (SLMs), offering a unique and timely solution for businesses. Altai's platform is designed to democratize AI access by empowering businesses to create and train their own custom SLMs through a revolutionary approach.

Here's how Altai's SLMs deliver a big impact:

• On-Premise (Company-Internal) and Secure Deployment: Altai understands the critical need for data sovereignty and privacy. Unlike many cloud-based LLMs, Altai allows for on-premise deployment, ensuring sensitive corporate data remains securely within the company's own infrastructure. This mitigates the risks of data leakage and simplifies compliance with strict regulations like GDPR, KVKK, and HIPAA.

• Superior Cost-Efficiency: Altai's focus on SLMs translates into significantly lower costs across the entire AI lifecycle.

  • Reduced Infrastructure & Energy: SLMs require substantially less computational power and memory for training and inference. This means lower hardware investments, reduced cloud bills (or no cloud bills with on-premise deployment), and a smaller carbon footprint, consuming up to 60% less energy than LLMs.
  • Faster & Cheaper Training/Fine-tuning: Altai's models can be trained and fine-tuned rapidly, in weeks, days, or even hours — a stark contrast to the months often required for LLMs. This dramatically reduces development costs.
  • Lower Operational Costs per Query: For high-volume tasks, the operational cost per query for an SLM can be over 100 times lower than for an LLM. This makes Altai's solutions highly budget-friendly for businesses with many AI interactions.

• Higher Accuracy in Targeted Tasks & Reduced Hallucinations: Altai develops domain-specific SLMs that are trained primarily on enterprise-specific documents and synthetically generated data derived from these documents.

  • This focused training on curated datasets allows SLMs to deeply understand the nuances, terminology, and factual landscape of a specific field. This leads to superior accuracy and reliability in targeted tasks compared to general-purpose LLMs.
  • This approach inherently mitigates hallucinations, as the model's knowledge scope is narrowed and grounded in verified company data, reducing the likelihood of generating false or irrelevant information.
  • Altai also leverages techniques like Retrieval-Augmented Generation (RAG) to provide SLMs with real-time access to up-to-date, transparent, and verifiable external or internal knowledge bases, further enhancing factual consistency and reducing hallucinations.

• Faster Inference Times: The smaller size and streamlined architecture of SLMs result in significantly faster response times. This low latency is crucial for real-time applications like customer service chatbots or interactive systems, enhancing user experience and operational fluidity.

• User-Friendly No-Code Platform: Altai offers an intuitive no-code interface. This democratizes AI creation, allowing businesses, including SMEs and non-technical users, to easily create and train custom AI-powered language models by simply uploading their documents, eliminating the need for specialized coding knowledge.

Conclusion​

The shift towards "right-sized AI" solutions is gaining significant traction in the enterprise world. Altai's focus on domain-specific SLMs directly addresses the critical pain points of high costs, inefficiency, and data security that plague traditional LLM deployments. By offering cost-efficient, high-accuracy, fast-inference, and secure on-premise solutions through a user-friendly no-code platform, Altai empowers businesses of all sizes to unlock the full, transformative potential of AI. This strategic approach not only enhances operational efficiency and reduces errors but also secures a tangible competitive advantage in today's dynamic market.

Citation​

Belcak, P., Heinrich, G., Diao, S., Fu, Y., Dong, X., Muralidharan, S., Lin, Y. C., & Molchanov, P. (2025). Small Language Models are the Future of Agentic AI. arXiv preprint. https://arxiv.org/abs/2506.02153

Originally published on Medium

Enterprises are eager to leverage large language models (LLMs) to drive automation, enhance decision-making, and streamline operations. However, adopting AI at scale comes with significant hurdles:

• Customization: Off-the-shelf models don't fit specific enterprise needs.
• Data Privacy: Cloud-hosted solutions pose compliance risks.
• Talent Gap: Hiring AI experts is costly and competitive.

At Altai, we're solving these challenges by providing enterprises with a no-code AI platform for effortless LLM customization and deployment, whether in the cloud or on-premises. Our vision is to democratize AI by combining the power of open-source models with high-quality synthetic dataset generation and enterprise-grade adaptation.

The Open-Source AI Opportunity​

Open-source LLMs like Llama and DeepSeek have made cutting-edge AI more accessible. However, enterprises require tailored solutions — domain-specific fine-tuning, secure on-prem deployment, and reliable data pipelines. As Y Combinator recently stated:

"Just as RedHat and YC W15 company GitLab built public companies around open-source projects, we are looking for companies built around open source AI models like DeepSeek and Llama. There is a huge opportunity to build startups that offer support and services to help people use open-source AI."

Altai seizes this opportunity by bridging the gap between open-source AI and real-world enterprise needs.

How Altai Works​

Our end-to-end workflow enables organizations to customize and deploy AI models without writing code:

1. Upload Documents → Process business-specific data.
2. Fine-Tune or Continual Pretrain → Adapt smaller LLMs to fit unique requirements.
3. Generate Synthetic Datasets → Use purpose-built LLMs and sophisticated algorithms to create and validate high-quality training data.
4. Instruction Fine-Tuning → Enhance models for enterprise-specific use cases by fine-tuning them on the validated synthetic dataset.
5. Deploy Securely → Deploy models on-prem or in the cloud.

By integrating automation at every step, Altai empowers organizations to build smarter, more relevant AI solutions in days rather than months.

Why Now?​

The demand for enterprise-ready AI is exploding. Companies struggle with:

• The need for domain-specific customization to get real business value from LLMs.
• Vendor lock-in from closed-source AI providers.
• Skyrocketing compute costs for training and inference.

Altai eliminates these barriers, making AI useful, adaptable, enterprise-ready, and cost-effective.

A Future Powered by Open, Customizable AI​

The AI landscape is shifting. Enterprises no longer need to rely on closed, black-box models from a handful of providers. With Altai, we're giving companies the tools to take control of their AI strategy — leveraging open-source innovation while ensuring customization, security, and scalability.

The future of AI is open, and Altai is here to make it work for you.

We're just getting started. If you're building with open-source AI and looking for enterprise-ready solutions, let's talk.

The Road Ahead​

We're building the foundation for scalable enterprise AI — a no-code, high-performance AI platform that empowers companies to own and control their models. If you're an enterprise looking to deploy AI without hiring a massive ML team, Altai is your answer.

Sign up for early access

Originally published on Medium

Introduction​

The rapid progress in Natural Language Processing (NLP) is mainly due to the development of large and complex neural network models. While these models perform very well, their size and high computational needs make them hard to use in real-world applications, especially for specific types of text. Many use cases — like semantic search and analyzing synthetic questions — could benefit from AI tools, but large models are often too costly and slow.

This study explores distilling the powerful multilingual BAAI/bge-m3 text embedding model into smaller versions with 8, 6, and 4 layers. The goal is to find the right balance between model size, speed, and retrieval performance using a synthetic dataset of contextual texts and generated questions. As NLP models become more complex, distillation helps make advanced AI more accessible and practical, especially for resource-limited needs.

1. Background​

1.1 What is Model Distillation?​

Model distillation — also known as knowledge distillation — is a technique used to compress large, powerful models by training smaller models to replicate their behavior. First introduced by Hinton et al. (2015), this approach allows a smaller "student" model to approximate the performance of a larger "teacher" model while using significantly fewer parameters. The result is faster inference, lower memory requirements, and reduced deployment costs — ideal for real-world applications that require scalability and efficiency.

At the heart of this process is the idea that the teacher model has already learned rich patterns and decision boundaries from extensive training data. Instead of training the student model solely on standard "hard" labels (like class IDs), distillation trains it on the "soft targets" produced by the teacher model. These soft targets — such as the probability distributions generated at the output layer — convey more nuanced information about how the teacher interprets the data. This helps the student not just learn the correct answers, but also gain insight into the reasoning behind the teacher's decisions.

In this work, the BAAI/bge-m3 model serves as the teacher. It is a state-of-the-art text embedding model, known for its effectiveness and versatility in a wide range of retrieval and representation tasks. It stands out in three important dimensions:

• Multi-Functionality: BAAI/bge-m3 supports dense retrieval, sparse retrieval (based on lexical matching), and multi-vector retrieval — all within a single model architecture.
• Multi-Linguality: It works across more than 100 languages, mapping them into a shared semantic space to enable both monolingual and cross-lingual search.
• Multi-Granularity: The model handles input texts of varying lengths, from short phrases to full-length documents — up to 8192 tokens — making it suitable for diverse use cases.

By distilling this teacher into smaller variants, the goal is to retain as much of its semantic power as possible, while greatly improving efficiency for deployment in real-world systems.

2. Distilled Variants​

The primary goal of this work was to create smaller, faster versions of the BAAI/bge-m3 model while retaining as much of its semantic understanding capabilities as possible, particularly for the contextual texts within the synthetic dataset. The motivation stems from the observation that while powerful models are highly effective, their large size can make them prohibitively expensive and slow for large-scale, low-latency applications.

The student models are:

• altaidevorg/bge-m3-distill-8l (8-layer distilled model)
• altaidevorg/bge-m3-distill-6l (6-layer distilled model)
• altaidevorg/bge-m3-distill-4l (4-layer distilled model)

The student models were derived from the BAAI/bge-m3 teacher model by reducing the number of model layers. This is a common and effective approach to creating smaller versions of transformer-based models.

Architecture: Transformer encoder with 4, 6, and 8 layers respectively for the distill-4l, distill-6l, and distill-8l models. The 8-layer model has a 366M parameter size, a significant reduction from the original 24-layer teacher. Other architectural parameters (hidden size, number of attention heads) are kept consistent with the layer configuration of the teacher model.

Training Data: Distillation used a mix of general text corpora and domain-specific contextual texts from the synthetic dataset (including question-answer pairs). This combination aimed to ensure both general understanding and adaptation to the target domain.

Use Cases: These models are intended for:

• Semantic search
• Information retrieval
• Document clustering
• Embedding components in RAG systems, especially for contextual texts in the synthetic dataset

Performance: The 8-layer model processed about 454 texts/sec on a T4 GPU — 2.5x faster than the teacher model (175 texts/sec). It also showed strong alignment with the teacher's output, scoring a Spearman Cosine of 0.965 and MSE of 0.006 on the test set.

Limitations:

• May lose some depth on complex tasks compared to the full teacher model.
• Might generalize less effectively to unrelated domains or unsupported languages.
• Could inherit biases from the teacher model or training data.

3. Experimental Setup​

To evaluate the performance of the distilled models, a robust experimental setup was designed, focusing on a semantic search task using a synthetic dataset composed of contextual texts and synthetically generated questions.

3.1 Dataset and Vector Search Process​

The dataset used in the evaluation process is a set of synthetically generated questions derived from contextual texts within the synthetic dataset. Each query is converted into a high-dimensional vector representation that captures its semantic meaning. This embedding is then used to search against the pre-indexed document embeddings in the Qdrant vector database. The system retrieves the top-k most semantically similar document IDs, where k is varied across multiple evaluation rounds (10, 20, 30, 40, 50).

3.2 Evaluation Metric: Precision@k​

Precision@k measures the proportion of overlapping items found within the top k results returned by two different models for the same query. When comparing Model A and Model B:

$$P@k_{A vs. B} = \frac{|{A\text{'s top-k docs}} \cap {B\text{'s top-k docs}}|}{k}$$

In this experimental context, Precision@k is used to compare the student models against each other (4L vs. 6L, 4L vs. 8L, 6L vs. 8L) for k values of 10, 20, 30, 40, and 50.

4. Results​

4.1 Performance (Precision@k Scores)​

Observations:

• The bge-m3-distill-6l and bge-m3-distill-8l models exhibit the highest similarity in their retrieval behavior across all tested values of k. P@k scores range from 0.700 at k=10 to 0.792 at k=50, indicating that their top retrieved documents are largely overlapping. This suggests that, despite having fewer layers, the 6-layer model closely mirrors the retrieval patterns of the more expressive 8-layer variant.

• In contrast, the bge-m3-distill-4l model shows noticeably lower alignment with both the 6L and 8L models. The P@10 score between the 4L and 8L models is 0.580, which gradually increases to 0.665 at P@50. This indicates that the 4-layer model diverges more in selecting the most relevant documents, particularly in the top-ranked results.

• A broader trend observed across all inter-model comparisons is that P@k values generally increase or stabilize as k increases. This means that while the most highly ranked results may vary more between models — especially between those with greater architectural differences like 4L vs. 8L — the overlap in retrieved documents grows when a larger set of top results is considered.

Efficiency Gains: The bge-m3-distill-8l model provides an empirically measured inference speedup of approximately 2.5x compared to the original 24-layer BAAI/bge-m3 teacher model, alongside substantial reductions in model size (366M parameters). The 6L and 4L variants are projected to offer even greater efficiency, with estimated speedups of approximately 3.1x and 4.0x respectively.

Conclusion​

This study on distilling the BAAI/bge-m3 model has resulted in the development of a series of more lightweight and computationally efficient embedding models. These distilled variants were designed to strike a balance between semantic performance and practical deployment needs such as speed and resource efficiency.

The 8-layer model emerges as a robust baseline, offering a 2.5x speedup over the original 24-layer teacher model with minimal loss in retrieval quality. It serves as a strong default option for most retrieval-based applications.

The 6-layer model demonstrates impressive alignment with the 8-layer variant, achieving a P@50 of 0.792, indicating that it preserves much of the retrieval behavior while being even more resource-efficient. This makes it an attractive choice for environments where performance needs to be balanced with greater throughput or tighter latency constraints.

The 4-layer model, while showing the largest deviation in top-k retrieval overlap (P@10 of 0.580 compared to the 8-layer model), offers the highest projected speedup of approximately 4.0x. Its lightweight architecture makes it especially well-suited for use cases where maximizing inference speed and minimizing computational footprint are paramount, even at the cost of some retrieval precision.

Together, these models provide a flexible set of options for embedding-based applications, enabling practitioners to make informed decisions based on their specific trade-off requirements between speed, scale, and retrieval fidelity.

Originally published on Medium

That's why we built llm-food — a FastAPI-based service that converts documents and URLs into clean, LLM-friendly Markdown. It supports batch processing, integrates with Google's Gemini for blazing-fast PDF OCR, and wraps everything in a CLI and Python client that doesn't make you cry.

tl;dr​

• Converts PDFs, DOCX, RTF, PPTX, and HTML/web pages into Markdown.
• Synchronous and async modes via HTTP API, CLI, and Python client.
• Batch PDFs go through Gemini Batch Prediction API for high throughput ($1 for ~6,000 pages).
• Tracks batch jobs in DuckDB.
• Fully dockerized. Optional auth. One-liner CLI.

Why?​

Extracting clean text from PDFs is still a mess.

You've probably seen dockling, marker, or pymupdf4llm. They're okay — but they're either slow, resource-hungry, or AGPL-licensed (ouch!). If you just want clean Markdown to fine-tune or RAG with your LLM, it's way more effort than it should be.

Enter Gemini Batch Prediction. It's fast and insanely cheap — $1 per 6,000 pages. But it's not dev-friendly.

So we wrapped it all up in a neat little microservice: llm-food.

What You Get​

Features​

• Convert:
  • PDFs (Gemini, pymupdf4llm, or pypdf)
  • DOC/DOCX (via mammoth)
  • RTF (via striprtf)
  • PPTX (python-pptx)
  • HTML or URLs (trafilatura)
• Run as:
  • FastAPI server (sync and async modes)
  • CLI (llm-food convert-file my.pdf)
  • Python client
• Async batch jobs with task tracking (DuckDB-based).
• Docker-ready, with optional Bearer token auth.
• Extensible and configurable.

Batch Mode: Fast, Cheap, Parallel​

PDFs get chunked and sent to Gemini's Batch API. You just upload files and give it a GCS path — we handle the queuing, status tracking, and Markdown export.

Other formats (DOCX, RTF, PPTX) are handled individually as background tasks.

How to Use It​

Install​

pip install 'llm-food[server]'          # Full server
pip install 'llm-food[server,pymupdf]'  # Optional: if you want pymupdf backend

Or just the client:

uv add llm-food

Start the Server​

llm-food-serve

API lives at http://localhost:8000/docs.

Want Docker?

docker build -t llm-food .
docker run -d -p 8000:8000 --env-file .env llm-food

CLI Usage​

llm-food convert-file ./paper.pdf
llm-food convert-url https://example.com
llm-food batch-create doc1.pdf doc2.pdf gs://my-bucket/outputs/
llm-food batch-status <task_id>
llm-food batch-results <task_id>

Set server URL/token via env:

export LLMFOOD_SERVERURL=http://my-host:8000
export LLMFOOD_APITOKEN=secret

Auth & Config​

Add a .env file to configure:

• GCS credentials
• Gemini API location/project
• File size limits
• PDF backend
• Auth token

Run locally or on a containerized server.

Roadmap​

• Support for transcription of media files (videos and audios)
• YouTube crawling and batch transcription
• Configurable automatic chunking

Final Thoughts​

Building a RAG pipeline, a search agent, or just need clean Markdown from messy enterprise docs? llm-food gets you there fast.

But let's be real: raw Markdown is just raw ingredients. We're cooking up an enterprise-grade engine that builds custom synthetic datasets and trains LLMs tailored to your docs. Join the waitlist or drop us a line if that's what you're hungry for.

AfterImage removes all of that.

What AfterImage does​

AfterImage is a no-code vision model trainer. You upload your images, label them, and click train. ALTAI handles the rest — model architecture selection, training infrastructure, optimization, and deployment.

The result: a production-ready REST API endpoint for your custom classifier.

Who it's for​

• Product teams who need visual inspection without hiring ML engineers
• Healthcare providers training diagnostic image classifiers on proprietary datasets
• Manufacturing companies building defect detection on their specific parts
• Defense and security teams with sensitive data that can't leave their environment

How it works​

1. Upload your images (JPG/PNG, any resolution)
2. Label each image with a class
3. Click Start Training
4. Get an API endpoint when training completes

No Python. No CUDA drivers. No infrastructure management.

Try it​

Create a free account and train your first model today.

The three failure modes​

1. Generic models on specific problems​

Most teams start with a foundation model (GPT-4, Gemini, Claude) and try to prompt-engineer their way to domain accuracy. It works for demos. It fails in production.

Your medical imaging data, your legal contracts, your manufacturing defects — these require models trained on your data, not on the internet.

2. Infrastructure that becomes a second job​

Building MLOps from scratch is a trap. Teams spend 80% of their time on infrastructure — managing training jobs, versioning models, scaling endpoints — instead of solving the actual business problem.

3. No path from prototype to production​

A notebook that works on your laptop is not a product. Most teams hit a wall when they try to scale from a POC to something their colleagues can actually use.

What works​

The enterprise AI projects that succeed share a pattern:

• Domain-specific data — they train on their own proprietary datasets
• Managed infrastructure — they use platforms that abstract away the ops burden
• API-first deployment — models are exposed as endpoints that existing systems can call

How ALTAI addresses this​

ALTAI is built around this pattern. You bring your data. We handle training infrastructure, optimization, and deployment. The output is a production-ready API endpoint — not a notebook, not a prototype.

See how it works →