AI & ML interests
AI models
Recent Activity
Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning
LoRA finetunes of Qwen3-VL-8B-Instruct, trained jointly on visual question answering and image captioning over overhead imagery. 13 experiments. All model repos are private.
Setup
| Item | Value |
|---|---|
| Base | Qwen3-VL-8B-Instruct |
| Method | LoRA (Unsloth), 4-bit base, bf16 compute |
| Targets | all linear layers, vision + language towers |
| Data | ~6.3k images, 1 caption + 2 QA each (~18.9k samples) |
| Caption prompts | sampled from a 30-prompt pool, so the model learns the task not one phrasing |
| Eval | fixed 1000-image draw = 2000 graded items, same seed for every model |
| Scoring | LLM-as-judge, 0-100 per item |
| References | stock Qwen3-VL-8B, plus an in-house finetuned model |
Winners are merged into the bf16 base and re-validated: adapters are evaluated on a 4-bit base, merged weights are not.
Configurations
| Model | r / alpha | Dropout | LR | Sched | Vision | Per image | Items | Change |
|---|---|---|---|---|---|---|---|---|
| Aspaklarya-001 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | reference recipe |
| Aspaklarya-002 | 32 / 64 | 0.05 | 1e-4 | cosine | trained | 1c + 2q | 18,864 | higher rank, lower LR |
| Aspaklarya-003 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 2c + 4q | 37,728 | 2x resampled data |
| Aspaklarya-004 | 16 / 16 | 0.0 | 2e-4 | linear | frozen | 1c + 2q | 18,864 | language-only |
| Aspaklarya-005 | 16 / 16 | 0.0 | 2e-4 | cosine | trained | 1c + 6q | 43,864 | VQA-heavy |
| Aspaklarya-006 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | batch 4 x accum 8 |
| Aspaklarya-007 | 8 / 8 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | low rank |
| Aspaklarya-008 | 16 / 16 | 0.0 | 1e-4 | linear | trained | 1c + 4q | 31,440 | more QA, lower LR |
| Aspaklarya-009 | 32 / 32 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | high rank |
| Aspaklarya-010 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | wider prompt pool |
| Aspaklarya-011 | 16 / 16 | 0.05 | 2e-4 | linear | trained | 1c + 2q | 18,864 | dropout, wd 0.05, clip 1.0 |
| Aspaklarya-012 | 16 / 16 | 0.0 | 2e-4 | linear | trained | 1c + 2q | 18,864 | as 011, no dropout |
| Aspaklarya-013 | 16 / 16 | 0.0 | 1e-4 | linear | trained | 1c + 2q | 18,864 | low LR, long schedule |
Constant everywhere: effective batch 32, warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.
Results
Best checkpoint per run, scored on 1000 images (2000 graded items). Overall = mean of caption and VQA. Right block = in-domain subset, closest to the target distribution.
| Model | Caption | VQA | Overall | In-dom. caption | In-dom. VQA | In-dom. overall |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B (base) | 65.68 | 56.01 | 60.84 | 68.37 | 58.94 | 63.66 |
| in-house reference | 65.18 | 73.34 | 69.26 | 71.88 | 83.62 | 77.75 |
| Aspaklarya-001 | 79.97 | 75.94 | 77.95 | 81.84 | 83.23 | 82.54 |
| Aspaklarya-011 | 79.92 | 75.78 | 77.85 | 81.85 | 83.51 | 82.68 |
| Aspaklarya-006 | 79.81 | 75.79 | 77.80 | 81.40 | 82.95 | 82.18 |
| Aspaklarya-005 | 79.55 | 75.74 | 77.64 | 81.37 | 82.37 | 81.87 |
| Aspaklarya-008 | 79.82 | 75.40 | 77.61 | 80.62 | 81.42 | 81.02 |
| Aspaklarya-003 | 79.55 | 75.52 | 77.53 | 81.23 | 82.18 | 81.71 |
| Aspaklarya-012 | 79.73 | 75.02 | 77.37 | 81.68 | 82.05 | 81.86 |
| Aspaklarya-007 | 79.33 | 75.33 | 77.33 | 81.13 | 80.82 | 80.97 |
| Aspaklarya-013 | 79.11 | 75.35 | 77.23 | 80.58 | 82.20 | 81.39 |
| Aspaklarya-009 | 78.53 | 75.85 | 77.19 | 80.01 | 82.34 | 81.18 |
| Aspaklarya-004 | 78.76 | 75.33 | 77.04 | 80.31 | 81.47 | 80.89 |
| Aspaklarya-002 | 79.36 | 74.33 | 76.85 | 80.70 | 80.63 | 80.67 |
| Aspaklarya-010 | 78.88 | 74.28 | 76.58 | 80.50 | 80.66 | 80.58 |
Every finetune beats both references: +17.1 overall over the base model, +8.7 over the in-house reference. Captioning gains most (66 to 80); VQA gains 20 points over base.
Caveat
Paired bootstrap over images puts the between-run standard error at ~0.45 overall, so the top eight runs are statistically tied — every confidence interval against the leader straddles zero. Doubling the eval set from 500 to 1000 images did not separate them, and it reshuffled the order: one run climbed six places purely on noise. Only the gaps to the two reference models are real.
Findings
- Optimization is saturated. LR, rank, alpha, dropout, weight decay, clipping, batch geometry and schedule were all swept; none beat the noise floor.
- The vision tower matters. Freezing it is the one ablation that consistently hurt.
- Extra QA per image does not help. 4-QA and 6-QA mixtures tie with 2-QA.
- Data is the remaining lever, not hyperparameters.