TTIC 31280 · Spring 2026 · Computer Vision

GRAFT: Region-Grounded
Fine-Tuning of SigLIP

Vision-language models trained on global image–caption contrastive objectives know what an image depicts, but not where. GRAFT teaches spatial grounding into a frozen SigLIP-B/16-384 using a tiny LoRA adapter (295K params, 0.14%) and a FILIP-style region loss on phrase→bounding-box pairs — lifting Flickr30k Pointing Game accuracy from a 14.74% baseline to 26.16% by eliminating train/eval representation mismatches.

PyTorch SigLIP-B/16-384 LoRA / PEFT FILIP region loss Flickr30k-Entities Florence-2 W&B

Pointing Game Gains
🎯
Pointing Game (test)
14.74 → 26.16%
+11.4pp
5-epoch run on a 28,467-phrase held-out test set (~48σ signal).
📊
Locked recipe (3-seed)
20.76 ± 0.41%
+6.02pp · 6× lower variance
Verified mean over seeds {0,1,2} — the statistically robust headline.
🪶
Trainable parameters
295K
0.14% of the backbone
LoRA rank-4 adapters on the vision tower's q/v projections; backbone frozen.
🤖
Auto-labeled (Florence-2)
23.35%
zero human bounding boxes
Machine-generated phrase→bbox supervision nearly matches human labels.

It Wasn't the Loss — It Was the Mismatch

The dominant gain came from aligning train-time and eval-time representations, not from loss design or model capacity.

Early versions trained patch features through the full self-attention layer but evaluated them through a MaskCLIP bypass, and represented phrases by a mean over all tokens while eval used only the EOS token. The gradient was sharpening the wrong vectors.

Installing the MaskCLIP bypass during training (vision side) flipped the model from below baseline to +3.94pp on the 3-seed mean. Switching the region loss to the EOS phrase feature (text side) added +2.08pp and collapsed seed variance 6×. Doubling the region-loss weight, adding LoRA capacity, and reformulating the patch aggregation all moved the mean within noise — the representation mismatch was the real lever.


Recipe
01

Frozen Backbone + LoRA

SigLIP-B/16-384 stays frozen (729 patches, 27×27 grid). A rank-4 LoRA adapter on the vision-tower q/v projections carries all trainable capacity.

rank 4 · α 16q_proj + v_proj295K params
02

Global + Region Loss

Sigmoid-SigLIP global loss on pooled image ↔ caption, plus a FILIP-style region loss between EOS phrase features and in-bbox patch features, max-pooled per image.

λ_region = 0.5FILIP max-poolbf16 autocast
03

Representation Alignment

MaskCLIP attention bypass on the vision side and EOS-token features on the text side make the training path byte-for-byte identical to the eval path. A best-PG snapshot restores the peak checkpoint.

MaskCLIP bypassEOS alignmentbest-PG snapshot
Florence-2 region annotations on a Flickr30k image
Florence-2 dense-region-caption annotations used to auto-generate phrase→bbox supervision for the human-free M_auto track.

Baseline → Locked Recipe → Multi-Epoch
VersionChangePointing GameΔ vs B0
B0Frozen SigLIP baseline14.74%—
v2+ MaskCLIP bypass during training+3.94% (3-seed mean)+3.94
v10+ EOS-aligned region loss + best-PG snapshot (locked, 3-seed)20.76 ± 0.41%+6.02
v115 epochs, per-epoch cosine restarts (single seed)26.16% (test)+11.42
M_autoFlorence-2 caption-grounded phrases (no human boxes)23.35%+8.61