SaylorTwift HF Staff
TASK.md: handoff for the next agent (state, artifacts, outstanding work, sharp edges)
7a5960d verified | # TASK: OpenCode-only RL on LFM2.5-2.6B — state and next steps | |
| *Written 2026-10-28 for an agent taking over from here.* | |
| ## TL;DR | |
| We reproduced the OpenCode-only RL run from | |
| [The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl) | |
| on `LiquidAI/LFM2.5-2.6B` with a LoRA adapter, end to end, entirely on HF | |
| infrastructure. **The pipeline is complete and battle-tested; the learning outcome | |
| was flat**: held-out pass@1 never separated from the base model (~12% ± 3% at n=100 | |
| through all 10 checkpoints), and held-out tool-call efficiency did not improve. | |
| The immediate open item is a clean **250-task final eval** (running now — see | |
| "Outstanding work" #1). | |
| ## Where everything lives | |
| | Artifact | Location | | |
| |---|---| | |
| | Code bundle (this project) | https://huggingface.co/SaylorTwift/lfm-opencode-lora-rl-code | | |
| | Recipe (architecture, versions, pitfalls) | `RECIPE.md` in that repo — **read it first** | | |
| | Base model | https://huggingface.co/LiquidAI/LFM2.5-2.6B | | |
| | Adapter repo (final + checkpoint-100…1000 + all eval JSONs) | https://huggingface.co/SaylorTwift/LFM2.5-2.6B-opencode-lora-RL (private) | | |
| | Learning curve | `eval/summary.json` in that repo | | |
| | Task data | `FineEnvs/SmolDataEnvs-harbor-train` / `-test` (pinned revisions, hash-checked by `prepare_tasks.py`) | | |
| | Article | https://huggingface.co/spaces/FineEnvs/multi-harness-rl | | |
| | Article's tutorial / env server | GitHub `adithya-s-k/FineEnvs` `05-multi-harness-rl/`; Space `FineEnvs/smoldataenv-multi-harness-harbor` | | |
| ## What was run and what it produced | |
| - **Smoke run** (20 steps) and **four full/partial training runs** across the | |
| debugging campaign; the last complete one (`6ae06e6f…`, h200x2, resumed from | |
| checkpoint-500) trained to step 1000 and pushed checkpoints 100–1000 + final. | |
| - **Eval worker** evaluated every checkpoint on 100 held-out tasks (base evaluated | |
| twice — both runs agree within 2%). | |
| - **Curve (pass@1, n=100, base = 12%):** | |
| 100:10, 200:13, 300:11, 400:12, 500:11, 600:8, 700:**15**, 800:10, 900:11, 1000:11. | |
| - **Tool calls on solved tasks vs base:** checkpoint-800 ×1.10, 900 ×1.04, 1000 ×1.02 — | |
| no held-out efficiency gain (training-pool tool calls dropped ~21% between two | |
| windows, but held-out cells say that was task-mix confounding). | |
| Config: LoRA r=32/α=64 on q/k/v/out_proj+w1/w2/w3, async GRPO (group of 8, OpenCode | |
| only), lr 1e-6, 1000 steps, temp 1.0/top_p 1.0/top_k 0, reward = correctness + ≤0.1 | |
| tool-efficiency bonus, output cap 4096, agent budget 32 calls. Full details in RECIPE.md. | |
| ## What is already proven (do not re-derive) | |
| - HF Sandboxes (`harbor[hf-sandbox]`, hf-sandbox backend, needs `docker_image` in | |
| task.toml — `prepare_tasks.py` patches it) run OpenCode unmodified. | |
| - Capture proxy records exact tokens+logprobs; importance ratio 0.997–1.01. | |
| - Merged-weight LoRA sync over NCCL: 136 ms/step at 2.6B. | |
| - Jobs-proxy exposure (`expose_public=[8000]` + `SPACE_HOST=$JOB_ID--8000.hf.jobs`) | |
| is stable for multi-day runs; the Gradio tunnel is not (it killed one run) and the | |
| Jobs proxy **strips Authorization**, hence the x-api-key seam patch in the entrypoints. | |
| - Watchdog (30-min stall → loud fail) + per-100-step checkpoint pushes (transformers v5 | |
| has no PushToHubCallback) + resume-from-repo all work. | |
| - trl must be **1.15.0** (1.16's tree packing rejects hybrid models) on **vllm 0.31.0**, | |
| with `kernels==0.16.0`. | |
| ## Outstanding work, in priority order | |
| 1. **Finish the clean 250-task final eval.** Job `6ae2026e3fb22a235dd409c2` is running | |
| (`run_eval.sh`, now fixed and hardened: test-only split + Jobs-proxy path). It writes | |
| `eval/final_v2_results.json`. A previous attempt produced `eval/final_results.json` | |
| but is **invalid** (env server had degraded after 11h; 247/250 cells failed per | |
| model — openenv client bug masks connection errors as `TypeError: object NoneType | |
| can't be used in 'await' expression`). Overwrite/ignore `eval/final_results.json`. | |
| If the job died, relaunch: `run_eval.sh` with env | |
| `ADAPTER_REPO=SaylorTwift/LFM2.5-2.6B-opencode-lora-RL, N_TASKS=250, EVAL_TAG=final_v2`, | |
| via `huggingface_hub.run_job(..., expose_public=[8000], flavor="h200", timeout="6h")`. | |
| 2. **Interpret the final eval** (250 tasks, ±2% error bars). If it confirms flat: | |
| the honest finding is *r=32 LoRA undertrains this model for agentic RL*. | |
| 3. **The natural follow-up run, in order of expected value:** | |
| a. **r=64 (or 128) LoRA** — one-line config change (`LoraConfig` in `train_lora.py`), | |
| plus `--max-lora-rank 64` wherever `--max-lora-rank 32` appears. Everything else identical. | |
| b. **Full fine-tune** (drop `peft_config`) — the article's actual setup; merged sync | |
| makes weight transfers ~5.2 GB/step (was fine at 136 ms/step). | |
| c. **SFT warm-start then RL** — the article's `FineEnvs/SmolDataEnvs-multiharness-sft` | |
| dataset is public and pre-tokenized for LFM2.5-2.6B; their runs showed SFT+RL > RL alone | |
| is the natural question. SFT took 1.5–6.5h of trainer time in the article. | |
| d. If pass@1 still won't move: raise training temperature exploration, check that the | |
| 100-task eval subset isn't saturated/hard-skewed (compare a random-100 draw), or | |
| evaluate a *bigger* base model on the same stack. | |
| 4. **Backfill efficiency metrics for checkpoints 100–700** — delete | |
| `eval/checkpoint-{100..700}_results.json` from the adapter repo and run | |
| `run_eval_watch.sh` for one pass (~2h total) if the complete efficiency curve matters. | |
| 5. **Housekeeping:** flip `SaylorTwift/LFM2.5-2.6B-opencode-lora-RL` to public when | |
| publishing; `export_adapter.py` regenerates the model card. The stalled-run artifacts | |
| (jobs `6ada06b…` through `6adf625f…`) can be inspected with `hf jobs inspect/logs`. | |
| ## Known sharp edges (all documented in RECIPE.md's pitfalls section) | |
| - HF Jobs proxy strips `Authorization` → session id must also ride `x-api-key` | |
| (patched into the OpenCode seam by every `run_*.sh` entrypoint). | |
| - Jobs proxy URL: `https://<JOB_ID>--8000.hf.jobs` (ports double-dashed). | |
| - `run_eval.sh`/`run_job.sh` must serve **only the splits they prepared** | |
| (`OPENENV_DATASETS`) — a missing split dir crashes the env server at boot. | |
| - Long-lived env servers degrade (websocket error loop after ~11h) — prefer fresh | |
| servers per eval; don't reuse the training job's server for the final eval. | |
| - openenv 0.10.0's MCP client masks connection failures with | |
| `TypeError: object NoneType can't be used in 'await' expression` — if you see that, | |
| it's a connection problem, not the code path you're reading. | |
| - `hf jobs logs` retains a bounded window; pull early, grep locally. | |
| ## Ops cheatsheet | |
| ```bash | |
| hf jobs ps --all # list jobs | |
| hf jobs logs SaylorTwift/<id> # fetch logs (grep locally, logs are big) | |
| hf jobs inspect SaylorTwift/<id> --format json # status incl. stage/message | |
| hf jobs cancel SaylorTwift/<id> | |
| hf download <repo> --include "eval/*" # grab results | |
| # Launch pattern: huggingface_hub.run_job(..., expose_public=[8000], timeout="6h", | |
| # flavor="h200", env={...}, secrets={"HF_TOKEN": token}) | |
| # (The CLI lacks expose_public; the eval path needs it because HF Jobs | |
| # auth-gated --expose breaks sandbox→proxy calls.) | |
| ``` | |