Same task. Less work.
What the skill changes
Before With skill · median per task
Before 2/3 · Wilson 95% CI 0.208–0.939; with skill 3/3 · Wilson 95% CI 0.439–1.000
Before 4/5 · Wilson 95% CI 0.376–0.964; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 0/3 · Wilson 95% CI 0.000–0.561; with skill 3/3 · Wilson 95% CI 0.439–1.000
Before 5/5 · Wilson 95% CI 0.566–1.000; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 3/3 · Wilson 95% CI 0.439–1.000; with skill 2/3 · Wilson 95% CI 0.208–0.939
Before 5/5 · Wilson 95% CI 0.566–1.000; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 2/3 · Wilson 95% CI 0.208–0.939; with skill 3/3 · Wilson 95% CI 0.439–1.000
Before 4/5 · Wilson 95% CI 0.376–0.964; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 0/3 · Wilson 95% CI 0.000–0.561; with skill 3/3 · Wilson 95% CI 0.439–1.000
Before 3/3 · Wilson 95% CI 0.439–1.000; with skill 2/3 · Wilson 95% CI 0.208–0.939
Before 5/5 · Wilson 95% CI 0.566–1.000; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 5/5 · Wilson 95% CI 0.566–1.000; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 4/5 · Wilson 95% CI 0.376–0.964; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 2/3 · Wilson 95% CI 0.208–0.939; with skill 3/3 · Wilson 95% CI 0.439–1.000
Before 0/3 · Wilson 95% CI 0.000–0.561; with skill 3/3 · Wilson 95% CI 0.439–1.000
Before 3/3 · Wilson 95% CI 0.439–1.000; with skill 2/3 · Wilson 95% CI 0.208–0.939
Before 5/5 · Wilson 95% CI 0.566–1.000; with skill 5/5 · Wilson 95% CI 0.566–1.000
Before 5/5 · Wilson 95% CI 0.566–1.000; with skill 5/5 · Wilson 95% CI 0.566–1.000
Try a skill
4 skills · 6 benchmarked actions
calculatorCalculator "3+3"−67%N=5/5
Try asking
- Calculate 1234 times 5
- calculate 1234 times 5
- use the calculator to divide 100 by 8
- what's 2 to the power of 10
calculate · N=5 seeds · 2026-10-22
skill version: 1.0.2 · SHA: b37d4c36335dfcf969967c5053762151415d685b
skill version: 1.0.2 · SHA: b37d4c36335dfcf969967c5053762151415d685b
skill version: 1.0.2 · SHA: b37d4c36335dfcf969967c5053762151415d685b
| Median | Before | With skill | Replay |
|---|---|---|---|
| Tokens | 184,681 | 61,837 | 0 |
| UI actions | 6 | 1 | 6 |
| LLM calls | 6 | 3 | 0 |
| Wall | 42 s | 60 s | 43 s |
Tokens saved: 123,150; median of paired seeds, N=5. Each pair is before minus with skill, including counted failures.
Full evidence, failures and traces →
contactsContacts: create Ada Lovelace−59%N=5/5
No clean demo yet for this action.
Try asking
- Add Ada Lovelace to my contacts with number +1 555 123 4567
- save Grace Hopper's number 555 0100
- create a contact for Alan Turing with email alan@example.com
- new contact Linus, phone +44 20 7946 0000
create_contact · N=5 seeds · 2026-10-22
skill version: 1.0.0 · SHA: b37d4c36335dfcf969967c5053762151415d685b
skill version: 1.0.0 · SHA: b37d4c36335dfcf969967c5053762151415d685b
skill version: 1.0.0 · SHA: b37d4c36335dfcf969967c5053762151415d685b
| Median | Before | With skill | Replay |
|---|---|---|---|
| Tokens | 889,687 | 367,882 | 0 |
| UI actions | 10 | 2 | 6 |
| LLM calls | 13 | 8 | 0 |
| Wall | 75 s | 84 s | 33 s |
Tokens saved: 343,499; median of paired seeds, N=5. Each pair is before minus with skill, including counted failures.
Full evidence, failures and traces →
clockClock: alarm 7:30 "Gym"−92%N=5/5
Try asking
- Set an alarm for 7:30 tomorrow morning called Gym
- wake me up at 6:45
- create an alarm at 22:00 labelled Meds
- set a 7am alarm
set_alarm · N=5 seeds · 2026-10-22
skill version: 1.0.1 · SHA: b37d4c36335dfcf969967c5053762151415d685b
skill version: 1.0.1 · SHA: b37d4c36335dfcf969967c5053762151415d685b
skill version: 1.0.1 · SHA: b37d4c36335dfcf969967c5053762151415d685b
| Median | Before | With skill | Replay |
|---|---|---|---|
| Tokens | 1,113,113 | 84,163 | 0 |
| UI actions | 14 | 1 | 6 |
| LLM calls | 20 | 4 | 0 |
| Wall | 1.9 min | 45 s | 22 s |
Tokens saved: 1,029,086; median of paired seeds, N=5. Each pair is before minus with skill, including counted failures.
Full evidence, failures and traces →
pro-expenseExpense: reimbursable rows from Markor−81%N=3/3
Try asking
- Go through the transactions in my_expenses.txt in Markor. Log the reimbursable transactions in the pro expense.
- log the reimbursable transactions from my_expenses.txt in pro expense
expense_add_multiple_from_markor · N=3 seeds · 2026-10-22
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
| Median | Before | With skill | Replay |
|---|---|---|---|
| Tokens | 2,283,598 | 440,495 | 0 |
| UI actions | 22 | 2 | not recorded |
| LLM calls | 25 | 9 | not recorded |
| Wall | 2.4 min | 2.3 min | 87 s |
Tokens saved: 2,057,001; median of paired seeds, N=3. Each pair is before minus with skill, including counted failures.
Full evidence, failures and traces →
pro-expenseExpense: add multiple rows−96%N=3/3
Try asking
- Add the following expenses into the pro expense: {rows}
- log these expenses in Pro Expense
- add the expenses from this list to pro expense
expense_add_multiple · N=3 seeds · 2026-10-22
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
skill version: 0.1.0 · SHA: a1980cff (feat/skillhub 1b3f8e66)
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
| Median | Before | With skill | Replay |
|---|---|---|---|
| Tokens | 1,530,202 | 60,438 | 0 |
| UI actions | 34 | 0 | 2 |
| LLM calls | 25 | 3 | 0 |
| Wall | 2.1 min | 1.8 min | 1.6 min |
Tokens saved: 1,408,794; median of paired seeds, N=3. Each pair is before minus with skill, including counted failures.
Full evidence, failures and traces →
pro-expenseExpense: add one row−65%N=3/3
Try asking
- Add the following expense into the pro expense: {rows}
- add an expense to pro expense
expense_add_single · N=3 seeds · 2026-10-22
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
skill version: 0.1.0 · SHA: a1980cff (feat/skillhub 1b3f8e66)
skill version: 0.1.0 · SHA: 38a33c2369123e8512141cd6cf2f573dfc5456d1
| Median | Before | With skill | Replay |
|---|---|---|---|
| Tokens | 1,752,326 | 621,898 | 0 |
| UI actions | 17 | 2 | 2 |
| LLM calls | 25 | 12 | 0 |
| Wall | 3.1 min | 3.0 min | 40 s |
Tokens saved: 1,224,179; median of paired seeds, N=3. Each pair is before minus with skill, including counted failures.
Full evidence, failures and traces →Replay: 0 tokens · 24/24 passed · no model.
Method & coverage
claude-haiku-5-5 · data date 2026-10-22. All 6 manifest tasks contribute their published current before/after_mcp arm counts. Every current counted failure is included. Historical pre-fix (supply-chain finding) rows are excluded.
19/24 · Wilson 95% CI 0.595–0.908 before; 23/24 · Wilson 95% CI 0.798–0.993 with skill. Paired McNemar: 5 for / 1 against, p=0.22, N=24 pairs.
KPI savings sum the task medians of paired-seed differences. Bars compare arm medians; N is before / with skill. UI actions are the agent’s own interactions. Tasks were chosen ad hoc; missing pairs are not invented.
calculator: 1/4 actions benchmarked · contacts: 1/4 actions benchmarked · clock: 1/7 actions benchmarked · pro-expense: 3/3 actions benchmarked. Two more skills are TBD.
Maintainer-run, not independent · claude-haiku-5-5 · Every failure is counted (intention-to-treat). · p not significant.