English | 中文

Quick start ·

Measured on SkillsBench

Mean verifier reward: +21.12 pp
Total tokens: -29.6% · Tool calls: -33.1%

SkillsBench is a benchmark designed to evaluate whether AI agents can effectively use Skills to complete professional tasks across diverse domains. Its purpose is to measure how much a model’s ability to solve complex real-world tasks improves when it is equipped with specialized Skills.

To evaluate performance in production-like environments where a large number of Skills are installed simultaneously, we adapted SkillsBench into a more realistic large-scale multi-Skill setting. In the original SkillsBench setup, each task is provided only with the specialized Skill associated with that task. In our modified setting, every task is evaluated in a global environment containing all 195 specialized Skills, while all other experimental conditions remain unchanged. This setting is intended to assess whether an agent can autonomously discover, select, and orchestrate the relevant Skills from a large installed Skill pool, and organize them into an effective workflow for completing complex tasks.

vibeskills v4.1.0 was benchmarked on SkillsBench (https://www.skillsbench.ai/) using DeepSeekV4Flash-VE and OpenHands as the baseline evaluation setup. Compared with the baseline without vibeskills, vibeskills increased the average task score by 21.12%, while reducing token consumption by 29.6% and tool calls by 33.1%.

_{Task quality: 39 to 57 full-score tasks; Lean Vibe scored higher on 23 tasks, Native on 4, with 55 ties.}

_{Resource use: 491.122M to 345.756M total tokens, with tool calls reduced from 9,954 to 6,664.}

Analysis of the logs from the original benchmark shows that VibeSkills achieves better task performance not by invoking more Skills.

Instead, it first clarifies the task objective and delivery requirements, then decomposes a complex task into several verifiable subtasks. It subsequently selects only a small number of truly relevant capabilities from a large pool of candidate Skills and executes them in an order determined by their dependencies. This helps reduce misunderstandings of the task, omitted steps, and incorrect Skill selection, thereby improving overall task performance.

In terms of token cost and tool usage, this workflow also eliminates a substantial amount of ineffective trial and error. Native agents are more likely to repeatedly invoke tools in unproductive directions, reread the same context, and redo previous work. In contrast, VibeSkills converges more quickly on the critical steps through clearer planning and pre-delivery checks. As a result, it not only improves task quality, but also significantly reduces tool-call loops and the token overhead caused by repeated context processing.

Study, public data, and reproduction ·

Task

Use public data to complete a reproducible classification experiment and deliver a data audit, statistical review, 4 result figures, a scientific report, and a 7-slide group-meeting deck.

The diagram shows what happened after the requirement and plan were approved: how the task was executed, what it produced, and how the result was checked.

The task used the L workflow and proceeded in order. During publication preparation, the configured folders on the same host contained more than 100 Skills. VibeSkills reviewed the candidates and their SKILL.md files, selected 7 for this task, and arranged the work into 5 groups and 10 work units. Those units covered environment setup, data audit, modeling, statistical review, figures, the report, and the slide deck.

After the work finished, VibeSkills ran 17 checks across the data, experiment results, figures, report, and slides. The task passed final acceptance after the required files, cross-deliverable consistency, and core reproduction all passed.

7 Skills selected · 5 work groups · 10 / 10 work units completed · 17 / 17 checks passed

%%{init: {"flowchart": {"curve": "monotoneX", "nodeSpacing": 18, "rankSpacing": 36}}}%%
flowchart LR
    subgraph DISC["Skill discovery"]
        direction TB
        A["Configured Skill folders<br/>100+ Skills"]
        B["Shortlist candidates<br/>Read SKILL.m