TASK.md

TASK: OpenCode-only RL on LFM2.5-2.6B — state and next steps

Written 2026-10-28 for an agent taking over from here.

TL;DR

We reproduced the OpenCode-only RL run from The ultimate guide to multi-harness RL on LiquidAI/LFM2.5-2.6B with a LoRA adapter, end to end, entirely on HF infrastructure. The pipeline is complete and battle-tested; the learning outcome was flat: held-out pass@1 never separated from the base model (~12% ± 3% at n=100 through all 10 checkpoints), and held-out tool-call efficiency did not improve. The immediate open item is a clean 250-task final eval (running now — see "Outstanding work" #1).

Where everything lives

Artifact	Location
Code bundle (this project)	https://huggingface.co/SaylorTwift/lfm-opencode-lora-rl-code
Recipe (architecture, versions, pitfalls)	RECIPE.md in that repo — read it first
Base model	https://huggingface.co/LiquidAI/LFM2.5-2.6B
Adapter repo (final + checkpoint-100…1000 + all eval JSONs)	https://huggingface.co/SaylorTwift/LFM2.5-2.6B-opencode-lora-RL (private)
Learning curve	eval/summary.json in that repo
Task data	FineEnvs/SmolDataEnvs-harbor-train / -test (pinned revisions, hash-checked by prepare_tasks.py)
Article	https://huggingface.co/spaces/FineEnvs/multi-harness-rl
Article's tutorial / env server	GitHub adithya-s-k/FineEnvs 05-multi-harness-rl/; Space FineEnvs/smoldataenv-multi-harness-harbor

What was run and what it produced

• Smoke run (20 steps) and four full/partial training runs across the debugging campaign; the last complete one (6ae06e6f…, h200x2, resumed from checkpoint-500) trained to step 1000 and pushed checkpoints 100–1000 + final.
• Eval worker evaluated every checkpoint on 100 held-out tasks (base evaluated twice — both runs agree within 2%).
• Curve (pass@1, n=100, base = 12%): 100:10, 200:13, 300:11, 400:12, 500:11, 600:8, 700:15, 800:10, 900:11, 1000:11.
• Tool calls on solved tasks vs base: checkpoint-800 ×1.10, 900 ×1.04, 1000 ×1.02 — no held-out efficiency gain (training-pool tool calls dropped ~21% between two windows, but held-out cells say that was task-mix confounding).

Config: LoRA r=32/α=64 on q/k/v/out_proj+w1/w2/w3, async GRPO (group of 8, OpenCode only), lr 1e-6, 1000 steps, temp 1.0/top_p 1.0/top_k 0, reward = correctness + ≤0.1 tool-efficiency bonus, output cap 4096, agent budget 32 calls. Full details in RECIPE.md.

What is already proven (do not re-derive)

• HF Sandboxes (harbor[hf-sandbox], hf-sandbox backend, needs docker_image in task.toml — prepare_tasks.py patches it) run OpenCode unmodified.
• Capture proxy records exact tokens+logprobs; importance ratio 0.997–1.01.
• Merged-weight LoRA sync over NCCL: 136 ms/step at 2.6B.
• Jobs-proxy exposure (expose_public=[8000] + SPACE_HOST=$JOB_ID--8000.hf.jobs) is stable for multi-day runs; the Gradio tunnel is not (it killed one run) and the Jobs proxy strips Authorization, hence the x-api-key seam patch in the entrypoints.
• Watchdog (30-min stall → loud fail) + per-100-step checkpoint pushes (transformers v5 has no PushToHubCallback) + resume-from-repo all work.
• trl must be 1.15.0 (1.16's tree packing rejects hybrid models) on vllm 0.31.0, with kernels==0.16.0.

Outstanding work, in priority order

1. Finish the clean 250-task final eval. Job 6ae2026e3fb22a235dd409c2 is running (run_eval.sh, now fixed and hardened: test-only split + Jobs-proxy path). It writes eval/final_v2_results.json. A previous attempt produced eval/final_results.json but is invalid (env server had degraded after 11h; 247/250 cells failed per model — openenv client bug masks connection errors as TypeError: object NoneType can't be used in 'await' expression). Overwrite/ignore eval/final_results.json. If the job died, relaunch: run_eval.sh with env ADAPTER_REPO=SaylorTwift/LFM2.5-2.6B-opencode-lora-RL, N_TASKS=250, EVAL_TAG=final_v2, via huggingface_hub.run_job(..., expose_public=[8000], flavor="h200", timeout="6h").
2. Interpret the final eval (250 tasks, ±2% error bars). If it confirms flat: the honest finding is r=32 LoRA undertrains this model for agentic RL.
3. The natural follow-up run, in order of expected value: a. r=64 (or 128) LoRA — one-line config change (LoraConfig in train_lora.py), plus --max-lora-rank 64 wherever --max-lora-rank 32 appears. Everything else identical. b. Full fine-tune (drop peft_config) — the article's actual setup; merged sync makes weight transfers ~5.2 GB/step (was fine at 136 ms/step). c. SFT warm-start then RL — the article's FineEnvs/SmolDataEnvs-multiharness-sft dataset is public and pre-tokenized for LFM2.5-2.6B; their runs showed SFT+RL > RL alone is the natural question. SFT took 1.5–6.5h of trainer time in the article. d. If pass@1 still won't move: raise training temperature exploration, check that the 100-task eval subset isn't saturated/hard-skewed (compare a random-100 draw), or evaluate a bigger base model on the same stack.
4. Backfill efficiency metrics for checkpoints 100–700 — delete eval/checkpoint-{100..700}_results.json from the adapter repo and run run_eval_watch.sh for one pass (~2h total) if the complete efficiency curve matters.
5. Housekeeping: flip SaylorTwift/LFM2.5-2.6B-opencode-lora-RL to public when publishing; export_adapter.py regenerates the model card. The stalled-run artifacts (jobs 6ada06b… through 6adf625f…) can be inspected with hf jobs inspect/logs.

Known sharp edges (all documented in RECIPE.md's pitfalls section)

• HF Jobs proxy strips Authorization → session id must also ride x-api-key (patched into the OpenCode seam by every run_*.sh entrypoint).
• Jobs proxy URL: https://<JOB_ID>--8000.hf.jobs (ports double-dashed).
• run_eval.sh/run_job.sh must serve only the splits they prepared (OPENENV_DATASETS) — a missing split dir crashes the env server at boot.
• Long-lived env servers degrade (websocket error loop after ~11h) — prefer fresh servers per eval; don't reuse the training job's server for the final eval.
• openenv 0.10.0's MCP client masks connection failures with TypeError: object NoneType can't be used in 'await' expression — if you see that, it's a connection problem, not the code path you're reading.
• hf jobs logs retains a bounded window; pull early, grep locally.

Ops cheatsheet

hf jobs ps --all                       # list jobs
hf jobs logs SaylorTwift/<id>          # fetch logs (grep locally, logs are big)
hf jobs inspect SaylorTwift/<id> --format json   # status incl. stage/message
hf jobs cancel SaylorTwift/<id>
hf download <repo> --include "eval/*"  # grab results
# Launch pattern: huggingface_hub.run_job(..., expose_public=[8000], timeout="6h",
#   flavor="h200", env={...}, secrets={"HF_TOKEN": token})
# (The CLI lacks expose_public; the eval path needs it because HF Jobs
#  auth-gated --expose breaks sandbox→proxy calls.)