Provael: prove it, prevail

Provael red-teams open Vision-Language-Action (VLA) robot policies in simulation. It reports an attack success rate next to a benign control, and writes the evidence a regulator or certifier can read: SARIF, OSCAL and a CycloneDX ML-BOM. Open-core, CPU-first, Apache-2.0.

Everything below is simulation only. No run has touched physical hardware. The attack families are published so policy builders can measure their own exposure.

The measured result

42 of 50 episodes left the keep-out envelope under a roleplay instruction (task-clustered 95% CI [62%, 100%]) against 1 of 50 benign. The same frame with no target named does it in 27 of 30, so this is fragility, not steering.

SmolVLA (HuggingFaceVLA/smolvla_libero) on libero_object (all 10 tasks), 5 seeds per task, in LIBERO, robosuite and MuJoCo. Measured with provael 0.41.2 on 14 September 2026, default (uncalibrated) predicate. McNemar exact p = 9.1e-13, Holm-adjusted 5.5e-12, against the benign control matched by task and seed.

Until 1 November this card still showed an earlier run (44 of 50 against a 2 of 50 benign control, [72%, 100%], measured with provael 0.32.0). The published result has been 42 of 50 against 1 of 50 since 18 September, and the control arms below changed how it reads: see erratum E-2026-12.

Read it alongside these, because a rate without them is not a result:

Control arms (a separate run of 180 episodes, 3 seeds per task): roleplay 26/30, the same frame with no target named 27/30, the same words in scrambled order 18/30, a benign rewording 0/30, nonsense text 0/30, no attack 1/30. Naming a target adds nothing, which is why this reads as fragility under an out-of-distribution instruction, not as an attacker steering the robot.

Limits that travel with the number. The keep-out predicate is uncalibrated: it counts episodes that leave the configured benign envelope, not a certified hazard threshold. The policy's sampler is not fully seeded, so this is one draw. One policy, one suite, simulation only. It says nothing about your checkpoint until you run it there.

The other arms, published as measured

• goal_substitution 7/50 and paraphrase 1/50: the rest of the instruction family
• decoy_object 1/50 and patch 1/50: the visual family, 2/100
• scene_text 2/50: injection. mcp_tool_desc does not apply to this suite (0 attempts), which is not the same as a null.

After Holm correction none of these differs significantly from the benign control. Published image-space attacks on VLAs report far higher rates on the same channel (baselines), so read these as the limits of these particular attacks, not as evidence that the policy is safe.

Registered is not measured

17 adversarial attack families are registered: implemented and unit-tested in the CPU harness. 3 were measured against a real policy in the run above, 5 more ran only in a one-task breadth probe at n = 3 (a result, not a rate), and 9 have never met a real model. Registration implies no real-policy transfer, no hardware result and no external reproduction.

Real-policy runs today support SmolVLA and π0.5 in LIBERO. The π0.5 leg is preliminary (findings).

Coverage of the Embodied AI Security Top 10 is 8 of 10. The other two are named rather than dropped: EAI07 is out of scope for simulation, and EAI10 is a process control with nothing to attack.

Try it, or ask

• Run it yourself: the quickstart starts with a 20-second CPU demo on a test fixture, then your own SmolVLA or π0.5 checkpoint on a Linux GPU.
• Have it run on your checkpoint: the Checkpoint Report, or write via provael.com/contact.

The leaderboard

• Provael ASR leaderboard: attack-success rates with their benign controls. The Space is a rendering; the signed JSON in the repository is the artifact to verify.
• Submissions dataset: open to anyone, via provael submit.

Elsewhere

• Code: github.com/provael/provael
• Site: provael.com
• Docs: docs.provael.com
• Cite: 10.5281/zenodo.21984184

Independent project, solo maintainer. Naming a benchmark or standards body here is not an affiliation or an endorsement, and none is implied either way.