Skip to content
Talk to us

Xupra Eval

One tool to test any robot AI, on any robot, in Japanese or English. Sakura is our core model.

In development · not yet released

Xupra Eval

The tool we use to run every test, from a single command.

  • Tests on real robots first, with simulation for quick checks
  • Works with vision-language-action models, LLM agents and classic controllers
  • Runs the same task in Japanese and English, side by side
  • Logs video, joint data and model calls so every run can be replayed
  • Embedded devices from Renesas
  • Open-source licence planned (Apache-2.0)
GitHub · coming soon Docs · coming soon
$ pip install xupra-eval # coming soon
$ xupra-eval run \
--bench genba-t1 --task "棚に並べて" \
--robot arm-01 --model <your-model> --trials 5
▸ preview build · results shown here are not real

What it is

Xupra Eval is the software we use to run every test in our lab. It loads a task, sends instructions to the model, controls the robot, records everything and scores the result. We plan to release it as open source so that anyone can repeat our tests.

What it will do

  • Run the same task on real hardware or in simulation
  • Connect to vision-language-action models, LLM-based agents and classic controllers through one interface
  • Give every instruction in Japanese and English and compare the results
  • Record video, joint states, model inputs and outputs for every trial
  • Score runs with clear, published rules, including safety outcomes
  • Export results as CSV and JSON, in the format our reports use

Hardware

Embedded devices from Renesas.

Task (YAML, JA/EN) Runner Model adapter Robot adapter Recorder and scorer Embedded devices from Renesas.

Licence

We plan to release Xupra Eval under the Apache License 2.0. Task definitions will be released under an open licence too, except for the unsafe prompts in Anzen Bench, which we keep private to protect the test.

Roadmap

  1. Internal use in our Tokyo lab
  2. Early access for outside labs, once the tool is ready
  3. Public open-source release with documentation
  4. Community tasks and robot adapters

Get involved

If you would like early access, want to add support for your robot, or have ideas for tasks, write to [email protected].

Test your model on real robots

Find out how your robot AI performs, in Japanese and English

Talk to us