# TASK: OpenCode-only RL on LFM2.5-2.6B — state and next steps

*Written 2026-10-28 for an agent taking over from here.*

## TL;DR

We reproduced the OpenCode-only RL run from
[The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl)
on `LiquidAI/LFM2.5-2.6B` with a LoRA adapter, end to end, entirely on HF
infrastructure. **The pipeline is complete and battle-tested; the learning outcome
was flat**: held-out pass@1 never separated from the base model (~12% ± 3% at n=100
through all 10 checkpoints), and held-out tool-call efficiency did not improve.
The immediate open item is a clean **250-task final eval** (running now — see
"Outstanding work" #1).

## Where everything lives

| Artifact | Location |
|---|---|
| Code bundle (this project) | https://huggingface.co/SaylorTwift/lfm-opencode-lora-rl-code |
| Recipe (architecture, versions, pitfalls) | `RECIPE.md` in that repo — **read it first** |
| Base model | https://huggingface.co/LiquidAI/LFM2.5-2.6B |
| Adapter repo (final + checkpoint-100…1000 + all eval JSONs) | https://huggingface.co/SaylorTwift/LFM2.5-2.6B-opencode-lora-RL (private) |
| Learning curve | `eval/summary.json` in that repo |
| Task data | `FineEnvs/SmolDataEnvs-harbor-train` / `-test` (pinned revisions, hash-checked by `prepare_tasks.py`) |
| Article | https://huggingface.co/spaces/FineEnvs/multi-harness-rl |
| Article's tutorial / env server | GitHub `adithya-s-k/FineEnvs` `05-multi-harness-rl/`; Space `FineEnvs/smoldataenv-multi-harness-harbor` |

## What was run and what it produced

- **Smoke run** (20 steps) and **four full/partial training runs** across the
  debugging campaign; the last complete one (`6ae06e6f…`, h200x2, resumed from
  checkpoint-500) trained to step 1000 and pushed checkpoints 100–1000 + final.
- **Eval worker** evaluated every checkpoint on 100 held-out tasks (base evaluated
  twice — both runs agree within 2%).
- **Curve (pass@1, n=100, base = 12%):**
  100:10, 200:13, 300:11, 400:12, 500:11, 600:8, 700:**15**, 800:10, 900:11, 1000:11.
- **Tool calls on solved tasks vs base:** checkpoint-800 ×1.10, 900 ×1.04, 1000 ×1.02 —
  no held-out efficiency gain (training-pool tool calls dropped ~21% between two
  windows, but held-out cells say that was task-mix confounding).

Config: LoRA r=32/α=64 on q/k/v/out_proj+w1/w2/w3, async GRPO (group of 8, OpenCode
only), lr 1e-6, 1000 steps, temp 1.0/top_p 1.0/top_k 0, reward = correctness + ≤0.1
tool-efficiency bonus, output cap 4096, agent budget 32 calls. Full details in RECIPE.md.

## What is already proven (do not re-derive)

- HF Sandboxes (`harbor[hf-sandbox]`, hf-sandbox backend, needs `docker_image` in
  task.toml — `prepare_tasks.py` patches it) run OpenCode unmodified.
- Capture proxy records exact tokens+logprobs; importance ratio 0.997–1.01.
- Merged-weight LoRA sync over NCCL: 136 ms/step at 2.6B.
- Jobs-proxy exposure (`expose_public=[8000]` + `SPACE_HOST=$JOB_ID--8000.hf.jobs`)
  is stable for multi-day runs; the Gradio tunnel is not (it killed one run) and the
  Jobs proxy **strips Authorization**, hence the x-api-key seam patch in the entrypoints.
- Watchdog (30-min stall → loud fail) + per-100-step checkpoint pushes (transformers v5
  has no PushToHubCallback) + resume-from-repo all work.
- trl must be **1.15.0** (1.16's tree packing rejects hybrid models) on **vllm 0.31.0**,
  with `kernels==0.16.0`.

## Outstanding work, in priority order

1. **Finish the clean 250-task final eval.** Job `6ae2026e3fb22a235dd409c2` is running
   (`run_eval.sh`, now fixed and hardened: test-only split + Jobs-proxy path). It writes
   `eval/final_v2_results.json`. A previous attempt produced `eval/final_results.json`
   but is **invalid** (env server had degraded after 11h; 247/250 cells failed per
   model — openenv client bug masks connection errors as `TypeError: object NoneType
   can't be used in 'await' expression`). Overwrite/ignore `eval/final_results.json`.
   If the job died, relaunch: `run_eval.sh` with env
   `ADAPTER_REPO=SaylorTwift/LFM2.5-2.6B-opencode-lora-RL, N_TASKS=250, EVAL_TAG=final_v2`,
   via `huggingface_hub.run_job(..., expose_public=[8000], flavor="h200", timeout="6h")`.
2. **Interpret the final eval** (250 tasks, ±2% error bars). If it confirms flat:
   the honest finding is *r=32 LoRA undertrains this model for agentic RL*.
3. **The natural follow-up run, in order of expected value:**
   a. **r=64 (or 128) LoRA** — one-line config change (`LoraConfig` in `train_lora.py`),
      plus `--max-lora-rank 64` wherever `--max-lora-rank 32` appears. Everything else identical.
   b. **Full fine-tune** (drop `peft_config`) — the article's actual setup; merged sync
      makes weight transfers ~5.2 GB/step (was fine at 136 ms/step).
   c. **SFT warm-start then RL** — the article's `FineEnvs/SmolDataEnvs-multiharness-sft`
      dataset is public and pre-tokenized for LFM2.5-2.6B; their runs showed SFT+RL > RL alone
      is the natural question. SFT took 1.5–6.5h of trainer time in the article.
   d. If pass@1 still won't move: raise training temperature exploration, check that the
      100-task eval subset isn't saturated/hard-skewed (compare a random-100 draw), or
      evaluate a *bigger* base model on the same stack.
4. **Backfill efficiency metrics for checkpoints 100–700** — delete
   `eval/checkpoint-{100..700}_results.json` from the adapter repo and run
   `run_eval_watch.sh` for one pass (~2h total) if the complete efficiency curve matters.
5. **Housekeeping:** flip `SaylorTwift/LFM2.5-2.6B-opencode-lora-RL` to public when
   publishing; `export_adapter.py` regenerates the model card. The stalled-run artifacts
   (jobs `6ada06b…` through `6adf625f…`) can be inspected with `hf jobs inspect/logs`.

## Known sharp edges (all documented in RECIPE.md's pitfalls section)

- HF Jobs proxy strips `Authorization` → session id must also ride `x-api-key`
  (patched into the OpenCode seam by every `run_*.sh` entrypoint).
- Jobs proxy URL: `https://<JOB_ID>--8000.hf.jobs` (ports double-dashed).
- `run_eval.sh`/`run_job.sh` must serve **only the splits they prepared**
  (`OPENENV_DATASETS`) — a missing split dir crashes the env server at boot.
- Long-lived env servers degrade (websocket error loop after ~11h) — prefer fresh
  servers per eval; don't reuse the training job's server for the final eval.
- openenv 0.10.0's MCP client masks connection failures with
  `TypeError: object NoneType can't be used in 'await' expression` — if you see that,
  it's a connection problem, not the code path you're reading.
- `hf jobs logs` retains a bounded window; pull early, grep locally.

## Ops cheatsheet

```bash
hf jobs ps --all                       # list jobs
hf jobs logs SaylorTwift/<id>          # fetch logs (grep locally, logs are big)
hf jobs inspect SaylorTwift/<id> --format json   # status incl. stage/message
hf jobs cancel SaylorTwift/<id>
hf download <repo> --include "eval/*"  # grab results
# Launch pattern: huggingface_hub.run_job(..., expose_public=[8000], timeout="6h",
#   flavor="h200", env={...}, secrets={"HF_TOKEN": token})
# (The CLI lacks expose_public; the eval path needs it because HF Jobs
#  auth-gated --expose breaks sandbox→proxy calls.)
```