References: gemma3_example.py (model class, GPTQ/W4A16 recipe) and gemma4_example.py (calibration dataset pattern).

Step 1: Choose the quantization method

GPTQ, via llm-compressor. Not AWQ.

• AutoAWQ is officially deprecated.

• Even setting that aside, AutoAWQ never supported Gemma3's architecture, which MedGemma is built on. llm-compressor has the same gap for AWQ specifically here — confirmed via two open GitHub issues describing outright failures.

Step 2: Set up your environment

Ran on RunPod (RTX A5000), using the instance's own pre-configured JupyterLab directly.

nvidia-smi   # check actual CUDA version first

pip3 uninstall torch torchvision -y
pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu128

python -c "import torch; print(torch.__version__); print(torch.cuda.is_available())"

Confirm True before continuing. Then install llmcompressor separately, with --no-deps (otherwise it silently pulls in a conflicting torch):

pip3 install llmcompressor==0.12.0 --no-deps

Then the other packages:

pip3 install transformers==5.10.1 compressed-tensors==0.17.1 requests==2.32.5 \
  pillow==11.0.0 zstandard==0.25.0 python-dotenv==1.2.2

Real final versions used: torch==2.11.0+cu128, torchvision==0.26.0+cu128, llmcompressor==0.12.0, compressed-tensors==0.17.1, transformers==5.10.1, requests==2.32.5, pillow==11.0.0, zstandard==0.25.0.

Set up your HF token before you need it — MedGemma is gated. Create a .env:

HF_TOKEN=hf_xxx

from dotenv import load_dotenv
load_dotenv()

Step 3: Load the model and processor

hf download google/medgemma-1.5-4b-it --local-dir ./medgemma-1.5-4b-it

from transformers import AutoProcessor, Gemma3ForConditionalGeneration

MODEL_ID = "./medgemma-1.5-4b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(MODEL_ID)
processor = AutoProcessor.from_pretrained(MODEL_ID)

Step 4: Build the calibration dataset manually

We used dataset="flickr30k", splits={...} but hit:

NotImplementedError: Label masking for vision datasets has not been implemented yet

This is gemma3_example.py's own approach — anyone following that example verbatim hits the same error. Fix: pretokenize calibration data manually instead (the gemma4_example.py pattern):

from datasets import load_dataset

NUM_CALIBRATION_SAMPLES = 32
MAX_SEQUENCE_LENGTH = 2048
BATCH_SIZE = 1

def get_calib_dataset(processor):
    ds = load_dataset("mit-han-lab/pile-val-backup", split=f"validation[:{NUM_CALIBRATION_SAMPLES * 10}]")
    def preprocess(example):
        return {"input_ids": processor.tokenizer.encode(example["text"].strip())[:MAX_SEQUENCE_LENGTH]}
    return (
        ds.shuffle(seed=42)
        .map(preprocess, remove_columns=ds.column_names)
        .filter(lambda ex: len(ex["input_ids"]) >= MAX_SEQUENCE_LENGTH)
        .select(range(NUM_CALIBRATION_SAMPLES))
    )

Text-only, even for a vision model — the vision tower is excluded from quantization anyway, so calibration only needs to exercise the language-model layers.

Step 5: Define the recipe

from llmcompressor.modifiers.gptq import GPTQModifier

recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        ignore=[
            "lm_head",
            r"re:model\.vision_tower.*",
            r"re:model\.multi_modal_projector.*"
        ],
    ),
]

Step 6: Run it

from llmcompressor import oneshot

oneshot(
    model=model,
    processor=processor,
    dataset=get_calib_dataset(processor),
    recipe=recipe,
    batch_size=BATCH_SIZE,
    shuffle_calibration_samples=False,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
)

Step 7: Verify the image actually reaches the model

from PIL import Image
import requests

messages = [
    {"role": "user", "content": [
        {"type": "text", "text": "Please describe what you see in this image\n"},
        {"type": "image"},
    ]},
]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)

image_url = "http://images.cocodataset.org/train2017/000000231895.jpg"
raw_image = Image.open(requests.get(image_url, stream=True).raw)
print(raw_image.size, raw_image.mode)

inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to(model.device)
print(inputs.keys())

Confirm pixel_values is in there. Real gotcha: it's easy to write processor(image=raw_image, ...) — singular — instead of images= (plural). Get this wrong and the image silently never reaches the model, but generation doesn't error — it just confidently describes something completely unrelated. Caught us for a bit. Check this before trusting any output.

Step 8: Run the real generation test

from compressed_tensors.offload import dispatch_model

dispatch_model(model)

# compile disabled — known issue: huggingface/transformers#38333
output = model.generate(**inputs, max_new_tokens=1024, disable_compile=True)
print(processor.decode(output[0], skip_special_tokens=True))

Confirm the description actually matches the image. Also re-ran this against a real chest X-ray — MedGemma's vision encoder is trained only on medical imagery, so a generic photo confirms plumbing but isn't a fair capability test.

Step 9: Save

QUANTIZED_MODEL_ID = MODEL_ID.rstrip("/").split("/")[-1] + "-W4A16-G128"
model.save_pretrained(QUANTIZED_MODEL_ID, save_compressed=True)
# do NOT call processor.save_pretrained() here — see Step 10

Step 10: Recover tokenizer/processor files from the original

Calling processor.save_pretrained() on this model renames extra_special_tokens to a non-standard model_specific_special_tokens key and drops added_tokens_decoder entirely — a documented transformers round-trip bug (also seen on Phi-4-mini). Copy the files directly from the original instead:

import shutil
from pathlib import Path

model_path = Path("./medgemma-1.5-4b-it")
quantized_model_path = Path(f"./{QUANTIZED_MODEL_ID}")

files_to_copy = [
    "tokenizer.json",
    "tokenizer_config.json",
    "special_tokens_map.json",
    "preprocessor_config.json",
    "chat_template.jinja",
]

for fname in files_to_copy:
    src = model_path / fname
    if src.exists():
        shutil.copy(src, quantized_model_path / fname)

Step 11: Patch the rope config

Gemma3's newer nested rope_parameters format (full_attention/ sliding_attention sub-dicts) isn't recognized by some downstream tooling expecting a flat schema with a top-level rope_type key. The key existed before, just nested — this restructures it, it doesn't add something absent. If you're following along in a notebook, add %%writefile patch_rope_config.py as the first line of the cell to write it straight to disk:

import json
import shutil
import sys
from pathlib import Path


def patch_config(config_path: str, dominant_rope_type: str = "default"):
    config_path = Path(config_path)
    backup_path = config_path.with_suffix(".json.bak")

    if not backup_path.exists():
        shutil.copy2(config_path, backup_path)
        print(f"Backed up original to {backup_path}")
    else:
        print(f"Backup already exists at {backup_path}, not overwriting it")

    with open(config_path, "r") as f:
        config = json.load(f)

    text_config = config.get("text_config")
    if text_config is None:
        print("ERROR: no 'text_config' block found — is this the right file?")
        sys.exit(1)

    rope_params = text_config.get("rope_parameters")
    if rope_params is None:
        print("ERROR: no 'rope_parameters' found under text_config")
        sys.exit(1)

    if "rope_type" in rope_params:
        print("'rope_type' key already present at top level — nothing to do")
        return

    new_rope_params = {"rope_type": dominant_rope_type}
    new_rope_params.update(rope_params)
    text_config["rope_parameters"] = new_rope_params

    with open(config_path, "w") as f:
        json.dump(config, f, indent=2)

    print(f"Patched {config_path}: added top-level rope_type='{dominant_rope_type}' "
          f"inside rope_parameters (full_attention/sliding_attention left untouched)")


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python patch_rope_config.py /path/to/config.json [rope_type]")
        sys.exit(1)
    path = sys.argv[1]
    rtype = sys.argv[2] if len(sys.argv) > 2 else "default"
    patch_config(path, rtype)

python patch_rope_config.py ./medgemma-1.5-4b-it-W4A16-G128/config.json

Keeps a .json.bak backup before touching anything, and is idempotent — safe to re-run if you're not sure whether it's already been applied.

Step 12: Fix the vision tower ignore list naming mismatch

A known llm-compressor bug (issue #1546): transformers >=4.52's multimodal refactor changed internal weight naming during model loading, but llm-compressor's quantization_config.ignore list is generated from the converted naming scheme without being reconciled against the actual saved safetensors key names. Left uncorrected, the ignore list matches no real tensor — the vision tower would silently get quantized despite the recipe saying otherwise.

import json

def fix_model_config(config_path):
    with open(config_path) as f:
        config = json.load(f)

    ignore_list = config["quantization_config"]["ignore"]

    new_ignore = []
    for entry in ignore_list:
        if entry.startswith("model.vision_tower."):
            fixed = entry.replace("model.vision_tower.", "vision_tower.vision_model.", 1)
            new_ignore.append(fixed)
        else:
            new_ignore.append(entry)  # e.g. "lm_head" stays as-is

    config["quantization_config"]["ignore"] = new_ignore

    with open(config_path, "w") as f:
        json.dump(config, f, indent=2)

    print("Sample before:", ignore_list[0])
    print("Sample after: ", new_ignore[0])

fix_model_config("medgemma-1.5-4b-it-W4A16-G128/config.json")

Step 13: Push to Hugging Face

If you set HF_TOKEN in your .env back in Step 2, huggingface_hub picks it up automatically. Otherwise, hf auth login first.

Don't use model.push_to_hub() — it calls save_pretrained() again internally, and since the model's already compressed, that throws KeyError: 'weight' (no plain weight key left to compress a second time). Upload the already-correct files directly instead:

from huggingface_hub import HfApi

REPO_ID = f"<your-hf-username-or-org>/{QUANTIZED_MODEL_ID}"
api = HfApi()
api.create_repo(REPO_ID, exist_ok=True)
api.upload_folder(folder_path=QUANTIZED_MODEL_ID, repo_id=REPO_ID)

Result: 8.6 GB in BF16 -> 5.2 GB quantized. Vision tower, multimodal projector, and lm_head stay unquantized, which is why it's not the full theoretical 75% reduction.

Model: confamnode/medgemma-1.5-4b-it-W4A16-G128

More technical AI blogs at ConfamNode