androidworld/pro-expense

Pro Expense bookkeeping as deterministic hard skills for the AndroidWorld Expense tasks; add expense rows from a list or from a Markor transactions file.

Draft 11.5× fewer tokens Score 16/100 Benchmark hard skill python filesstorage benchmark reference

Before / after

Benchmark details ↓
Tokens saved4,689,974Paired medians · 3/3 tasks · N=9
UI actions saved67Paired medians · 3/3 tasks · N=9
Pass · before → with skill5/9 → 8/9Current arms · failures included

Before With skill · median tokens per task

Expense: reimbursable rows from Markor Expense: reimbursable rows from Markor. Median tokens: before 2,283,598, with skill 440,495. N=3 before / 3 with skill. 2,283,598440,495 −81% tokensN=3/3 N=3 before / 3 with skill
Before 0/3 · Wilson 95% CI 0.000–0.561; with skill 3/3 · Wilson 95% CI 0.439–1.000
Expense: add multiple rows Expense: add multiple rows. Median tokens: before 1,530,202, with skill 60,438. N=3 before / 3 with skill. 1,530,20260,438 −96% tokensN=3/3 N=3 before / 3 with skill
Before 2/3 · Wilson 95% CI 0.208–0.939; with skill 3/3 · Wilson 95% CI 0.439–1.000
Expense: add one row Expense: add one row. Median tokens: before 1,752,326, with skill 621,898. N=3 before / 3 with skill. 1,752,326621,898 −65% tokensN=3/3 N=3 before / 3 with skill
Before 3/3 · Wilson 95% CI 0.439–1.000; with skill 2/3 · Wilson 95% CI 0.208–0.939

maintainer-run, not independent · claude-haiku-5-5 · 2026-10-22. Every failure is counted (intention-to-treat).

Demo: expense_add_multiple · HOME → launch → action

Control Pro Expense on Android with an AI agent — track expenses

Pro Expense (com.arduia.expense) bookkeeping as deterministic hard skills for the AndroidWorld Expense tasks. Status: active for the three add tasks; the delete and de-duplicate tasks and the gallery (OCR) variant are not authored yet.

Attribution

Task names, goals and the evaluator contract come from google-research/android_world at commit 2655e27cd090f97fbf2ccaecf759057bec0a8460, licensed under Apache-2.0 (see LICENSE). The UI steps were written from the running app on the AndroidWorld Pixel 6 / API 33 image, never from the evaluator internals; the skill takes the task's values as params and hard-codes nothing from a task instance.

Actions

ActionAndroidWorld taskParams
expense_add_multipleExpenseAddMultiple (complexity 6)rows: JSON list of {name, amount, category, note}
expense_add_singleExpenseAddSingle (complexity 1.2)rows: one-element list
expense_add_multiple_from_markorExpenseAddMultipleFromMarkor (complexity 6)none; reads Documents/Markor/my_expenses.txt and logs the rows whose note says Reimbursable

amount is dollars as text ("12.50"); the app stores cents. amount_dollars and category_name (the AndroidWorld task wording) are accepted as aliases, and a row with no amount or no category is refused rather than saved as 0.00 / Others. category is one of Others, Income, Food, Housing, Social, Entertainment, Transportation, Clothes, Health Care, Education, Donation.

ghost skill run androidworld/pro-expense expense_add_multiple --device <serial> \
  --param 'rows=[{"name": "Train ticket", "amount": "23.40", "category": "Transportation", "note": "Client visit"}]'

How it works

  1. open_pro_expense launches the app and finishes the first-run language and currency pages when they show.
  2. helpers:add_rows taps the add button for every row, types name and amount, picks the category on the horizontal strip (scrolling it into view), types the note and saves.
  3. The end-state validator reads the app's own SQLite table over adb (adbd runs as root on the emulator image) and checks that every requested row is present with the exact cents, category id and note, normalizing the caller input again rather than trusting the returned rows. If SQLite is inaccessible, it opens saved entries from today's expense log and independently reads the existing editor's full name, amount, note and checked category. It leaves that editor without pressing UPDATE.

Bench

ghost skill bench androidworld/pro-expense --androidworld ExpenseAddMultipleFromMarkor --bridge-cmd "<aw_bridge command>" replays the action on an AndroidWorld emulator and lets AndroidWorld's task.is_successful judge it; results land in bench/androidworld.json (display only) and in signed run artifacts with a bench block.

Failure modes

  • The category strip did not scroll to the wanted name within 12 swipes: the strip order changed with an app update; fix CATEGORIES in helpers.py.

  • my_expenses.txt is not in the Markor notebook: the Markor file path differs from AndroidWorld's Documents/Markor; pass the task through the bench so initialize_task writes it.

  • The unrooted UI validator searches at most eight log screens and twelve category-strip views. Missing entries, ambiguous/unchecked categories, unexpected screens or mismatched stored fields fail validation.

  • App field limits are checked before row entry: name 20 characters, amount 10 characters, note 100 characters. Oversized inputs are rejected instead of silently truncated by Android.

  • Rows with no recognized amount/category, an empty name, a malformed amount, or an unknown category are refused. Since a1980cff, amount_dollars and category_name are recognized aliases. Unrelated extra keys are ignored, not categorically rejected.

  • The pre-fix ab3 finding saved 0.00 / Others and returned success for task-worded inputs; agents repaired those rows manually. Historical results remain labelled in the bench card, with original validity flags and notes intact.

  • Historical AddSingle seed 31 failed across three 20-second launch waits; a later AddMultiple replay seed 30 also missed the editor within 10 seconds after the Add FAB. The navigation fix now waits up to 60 seconds per launch attempt, confirms Pro Expense is foreground before tapping the FAB, and waits up to 30 seconds for the editor. If the editor is still absent and the foreground home FAB is visible, it re-taps that FAB once and waits another 30 seconds. A late editor suppresses the retry. Row entry and Save are never retried by this recovery; validators are unchanged. Live confirmation is pending ghost-bencher replay.

  • Database validation checks membership of each requested row, not a before/after insertion count. A pre-existing identical row can satisfy it; it does not prove the number of duplicates added.

Compatibility

The broad manifest fingerprint for com.arduia.expense is not proof of general compatibility. Rooted AndroidWorld uses SQLite validation; unrooted Samsung Android 13 uses the saved-entry UI fallback. Both checks verify row membership rather than insertion counts. Consult current digest-bound evidence and the device grid for exercised app versions, screens and locales.

Try a sample prompt or run an action by name. Replace any parameters in braces with your own values.

expense_add_multiple

Play demo

Try asking

  • Add the following expenses into the pro expense: {rows}
  • log these expenses in Pro Expense
  • add the expenses from this list to pro expense
ghost skill run androidworld/pro-expense expense_add_multiple

expense_add_single

Play demo

Try asking

  • Add the following expense into the pro expense: {rows}
  • add an expense to pro expense
ghost skill run androidworld/pro-expense expense_add_single

expense_add_multiple_from_markor

Play demo

Try asking

  • Go through the transactions in my_expenses.txt in Markor. Log the reimbursable transactions in the pro expense.
  • log the reimbursable transactions from my_expenses.txt in pro expense
ghost skill run androidworld/pro-expense expense_add_multiple_from_markor
Benchmark details

AndroidWorld before/after

maintainer-run, not independent
Model: claude-haiku-5-5 · Date: 2026-10-22

Same task across three arms. Skill replay runs without a model. Medians include every valid episode, including failures.

Expense: reimbursable rows from Markor

N=3 seeds · 2026-10-22

Arm / provenancePass a/b · Wilson 95% CITokens, median [q1–q3]LLM calls, medianUI actions, medianWall, median
Agent aloneclaude-haiku-5-5 · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
0/3 · Wilson 95% CI 0.000–0.5612,283,598[1,827,766–2,480,248.5]25222.4 min
Skill replayno model · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
3/3 · no CI0[no quartiles]——87 s
Agent + skillclaude-haiku-5-5 · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
3/3 · Wilson 95% CI 0.439–1.000440,495[333,546–517,689]922.3 min
  • Agent alone: void (3 episodes discarded). net.gsantner.markor was uninstalled by the before-30 episode; before-31/32 and all after_mcp episodes ran without the task app. Superseded by run ab1 (tools/bench/agent_arm.py, task-app guard).model not recorded · 2026-10-22 · Skill 0.1.0 · SHA 38a33c23
  • Agent + skill: void (2 episodes discarded). net.gsantner.markor was uninstalled by the before-30 episode; before-31/32 and all after_mcp episodes ran without the task app; after_mcp also could not load hub skills (find_skill pointed at the v1 run_action runner). Superseded by run ab1 (tools/bench/agent_arm.py, task-app guard).model not recorded · 2026-10-22 · Skill 0.1.0 · SHA 38a33c23

Agent + skill vs agent alone: 3 paired seeds; 3 fail → pass, 0 pass → fail; McNemar exact p = 0.25. Median tokens saved: 2,057,001.

Expense: add multiple rows

N=3 seeds · 2026-10-22

Arm / provenancePass a/b · Wilson 95% CITokens, median [q1–q3]LLM calls, medianUI actions, medianWall, median
Agent aloneclaude-haiku-5-5 · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
2/3 · Wilson 95% CI 0.208–0.9391,530,202[1,488,018–1,915,058.5]25342.1 min
Skill replayno model · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
3/3 · Wilson 95% CI 0.439–1.0000[0–0]021.6 min
Agent + skillclaude-haiku-5-5 · 2026-10-22
Skill 0.1.0 · SHA a1980cff
3/3 · Wilson 95% CI 0.439–1.00060,438[60,403.5–90,923]301.8 min

pre-fix (supply-chain finding)

Agent + skill on the pre-fix skill (supply-chain finding): 1/3 passed · claude-haiku-5-5 · 2026-10-22 · Skill 0.1.0 · SHA 38a33c23. the skill saved rows given with the task keys amount_dollars/category_name as 0.00/Others and still returned success (helpers.py add_rows defaults); the agent repaired them by hand. Agent + skill here measures that bug. Seed 32 counted as FAIL (intention-to-treat; cut by the run budget while repairing).

Agent + skill on the pre-fix skill (supply-chain finding): historical episodes
ArmSeedModelResultTokensTrace
Agent + skill30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass1,533,383
Agent + skill31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23fail2,357,761
Agent + skill32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23fail (intention-to-treat; source marked invalid): agent: token cap 1232905void

Skill replay vs agent alone: 3 paired seeds; 1 fail → pass, 0 pass → fail; McNemar exact p = 1. Median tokens saved: 1,530,202.

Agent + skill vs agent alone: 3 paired seeds; 1 fail → pass, 0 pass → fail; McNemar exact p = 1. Median tokens saved: 1,408,794.

Expense: add one row

N=3 seeds · 2026-10-22

Arm / provenancePass a/b · Wilson 95% CITokens, median [q1–q3]LLM calls, medianUI actions, medianWall, median
Agent aloneclaude-haiku-5-5 · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
3/3 · Wilson 95% CI 0.439–1.0001,752,326[1,725,074–1,799,201.5]25173.1 min
Skill replayno model · 2026-10-22
Skill 0.1.0 · SHA 38a33c23
3/3 · Wilson 95% CI 0.439–1.0000[0–0]0240 s
Agent + skillclaude-haiku-5-5 · 2026-10-22
Skill 0.1.0 · SHA a1980cff
2/3 · Wilson 95% CI 0.208–0.939621,898[340,836.5–755,340]1223.0 min

pre-fix (supply-chain finding)

Agent + skill on the pre-fix skill (supply-chain finding): 3/3 passed · claude-haiku-5-5 · 2026-10-22 · Skill 0.1.0 · SHA 38a33c23. the skill saved rows given with the task keys amount_dollars/category_name as 0.00/Others and still returned success (helpers.py add_rows defaults); the agent repaired them by hand. Agent + skill here measures that bug.

Agent + skill on the pre-fix skill (supply-chain finding): historical episodes
ArmSeedModelResultTokensTrace
Agent + skill30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass1,914,855
Agent + skill31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass1,136,564
Agent + skill32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass917,944

Agent + skill vs agent alone: 3 paired seeds; 0 fail → pass, 1 pass → fail; McNemar exact p = 1. Median tokens saved: 1,224,179.

Skill replay vs agent alone: 3 paired seeds; 0 fail → pass, 0 pass → fail; McNemar exact p = 1. Median tokens saved: 1,752,326.

Failed episodes

  • Expense: reimbursable rows from Markor · Agent alone · seed 30 · claude-haiku-5-5
  • Expense: reimbursable rows from Markor · Agent alone · seed 31 · claude-haiku-5-5
  • Expense: reimbursable rows from Markor · Agent alone · seed 32 · claude-haiku-5-5
  • Expense: add multiple rows · Agent alone · seed 31 · claude-haiku-5-5
  • Expense: add one row · Agent + skill · seed 31 · claude-haiku-5-5
Episodes (24)
TaskArmSeedModelPassLLM callsTokensUI actionsWallTrace
Expense: reimbursable rows from MarkorAgent + skill30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass11595k42.4 min
Expense: reimbursable rows from MarkorAgent alone30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23fail252.68M222.3 min
Expense: reimbursable rows from MarkorAgent + skill31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass4227k01.6 min
Expense: reimbursable rows from MarkorAgent alone31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23fail252.28M242.4 min
Expense: reimbursable rows from MarkorAgent + skill32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass9440k22.3 min
Expense: reimbursable rows from MarkorAgent alone32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23fail251.37M172.5 min
Expense: add multiple rowsAgent + skill30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA a1980cffpass360k02.0 min
Expense: add multiple rowsSkill replay30replay2026-10-22 · Skill 0.1.0 · SHA 38a33c23pass0021.8 min
Expense: add multiple rowsAgent alone30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass251.45M342.1 min
Expense: add multiple rowsAgent + skill31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA a1980cffpass4121k01.8 min
Expense: add multiple rowsSkill replay31replay2026-10-22 · Skill 0.1.0 · SHA 38a33c23pass0021.5 min
Expense: add multiple rowsAgent alone31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23fail251.53M152.1 min
Expense: add multiple rowsAgent + skill32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA a1980cffpass360k01.8 min
Expense: add multiple rowsSkill replay32replay2026-10-22 · Skill 0.1.0 · SHA 38a33c23pass0021.6 min
Expense: add multiple rowsAgent alone32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass252.30M352.8 min
Expense: add one rowAgent + skill30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA a1980cffpass360k061 s
Expense: add one rowSkill replay30replay2026-10-22 · Skill 0.1.0 · SHA 38a33c23pass00245 s
Expense: add one rowAgent alone30claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass251.70M233.8 min
Expense: add one rowAgent + skill31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA a1980cfffail (intention-to-treat): episode capped after the skill launch timeout16889k123.0 min
Expense: add one rowSkill replay31replay2026-10-22 · Skill 0.1.0 · SHA 38a33c23pass00240 s
Expense: add one rowAgent alone31claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass251.75M152.1 min
Expense: add one rowAgent + skill32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA a1980cffpass12622k23.6 min
Expense: add one rowSkill replay32replay2026-10-22 · Skill 0.1.0 · SHA 38a33c23pass00238 s
Expense: add one rowAgent alone32claude-haiku-5-52026-10-22 · Skill 0.1.0 · SHA 38a33c23pass201.85M173.1 min

Display only: points come from signed run artifacts. AndroidWorld @ 2655e27c.