How skills are scored
Every skill carries two separate badges. Trust says who published it and how it got into the hub. Quality says whether it works and how well it is documented. This page is the whole quality formula, scoring v1.3: a score from 0 to 100.
auto-login runner: beta. Action state badges declare prerequisites and transitions: “Requires: logged in”, “Sets: logged in”, or “Clears: logged in”. They do not prove a session exists or add scoring points. Deterministic probes check the current device; signed passing runs remain the evidence.
Trust
| Official | Written and maintained by the Ghost team under skillhub/ghost/. Says who wrote it, not that anyone else verified it; the Reliability card shows the runs. Installs silently. |
| Verified | Community skill merged into the hub repo through a recorded review. |
| Benchmark | Reference skill imported from a benchmark suite (AndroidWorld) by the maintainers. Not reviewed community code; most are scaffolds with no authored steps yet. |
| Unverified | No recorded review: installed from a URL outside the hub repo, or merged without one. The CLI asks before installing. |
| Quarantined | Flagged by the registry. Install is blocked unless forced. |
Quality tiers
A skill needs both the points and every hard gate of a tier to hold it. Tiers are a ladder: each one includes the gates below it.
-
Draft
0 points- skill.yaml validates
- Static scan is clean
-
Runs
10 points- README present
- One signed self-run artifact (fingerprint hash, app version, per-step result)
-
Proven
35 points- Your own signed passing run of the current version on every action
- A demo clip of 3 MB or less (webm, mp4 or gif)
- 3 validator units on every action (custom_python or a parameterised text check = 2, a bare presence check = 1)
-
Hardened
60 points- One passing run signed by another account, on a device fingerprint the author has not recorded on
- 2 or more OEM or Android-major combinations
- Bench on 2 or more models
- 10 or more community runs with Wilson lower bound (95%) of 0.80 or higher
- "Known failure modes" section in the docs
- Re-verified in the last 90 days
-
Reference
90 points- Hardened for 60 or more continuous days
- Curator review
- No open P1 bug older than 14 days
- Author at Contributor tier or above
- Capped at about 5% of Hardened skills per category; re-reviewed every 180 days or on a major target-app release
Where points come from
Each category has a cap, and the caps add up to 100, so no single axis can carry a skill up the ladder.
| Category | Cap | Earned by |
|---|---|---|
| Docs and packaging | 20 | Manifest +3, Readme +3, Known failure modes +4, Demo clip +4, Demo for every action +3, Before / after pair (clip + metrics) +3 |
| Self-evidence | 30 | First self-run +4, Bench on one model +4, Second device of your own (every action) +4, Validators (per action) +2 each per action, up to +8 (weights: custom python x2, parameterised text x2, default x1), Self-reliability (your own pass rate) up to +10, Wilson lower bound over your own verified runs once 3 exist on every action |
| Independent verification | 30 | First independent +12, Second device class +8, Third device class +4, Bench second model +4, Bench third model +2 |
| Community usage (90 days) | 10 | Wilson z +1.96, Min runs +10, Window days +90 |
| Maintenance | 10 | Reverified 30d +6, Bug response 7d +2 each, up to +4 |
| Downloads and likes | 0 | Shown on the page, never scored. |
Self-reliability. Up to 10 points inside Self-evidence for the pass rate of your own verified runs, pass or fail: the Wilson lower bound (95%) of passed over total, times 10, rounded. It starts counting at 3 recorded self-runs, so a single lucky run earns nothing and a skill that fails often loses the points on its own evidence.
Validators are scored per action. The action with the fewest validator units sets the number: +2 per unit up to +8, where custom python counts x2, parameterised text counts x2, default counts x1. Ten trivial checks on one action cannot stand in for the rest.
Rules in the scoring file
Verbatim from skillhub/scoring/v1.json (v1.3); the registry builder applies exactly these.
- Evidence is classified by the SIGNING KEY, never by the label on the artifact: self (recorded by the author), maintainer (signed with the author's or a maintainer-tier key, whatever verifier name it carries), ci (a platform verifier with its own key; none exists yet), independent (a SIGNERS key that is neither the author's nor maintainer-tier).
- Only independent evidence earns independent points. Maintainer-signed replays and CI reproductions are shown on the card and feed maintenance, never the independent category.
- A run counts only for the code on disk: its recorded version must equal the current version and, when it carries skill_sha256, that digest must equal the current shipped-file digest. Runs on older versions are shown as previous versions and earn nothing.
- Run credit is per action: self, maintainer, ci and independent run counts are the minimum over the skill's actions of passing current runs that exercised that action. A skill with eight actions and runs for one has zero.
- Validators earn points only on actions that have at least one passing current run, and never for a skill in status draft; the weakest action decides.
- An independent device class counts only for a fingerprint the author has not recorded on; one report per fingerprint.
- Maintenance (re-verified) comes from maintainer, ci or independent passing runs of the current version, never from the author's own runs; full value within 30 days, then a linear decay to zero at 60 days measured against the real build date.
- Bench models are counted from verified artifacts that carry a bench block with an agent model; declarations in bench.yaml earn nothing. The deterministic replay arm is not a model.
- Self-reliability needs 3 or more of your own verified current runs on EVERY action (the action with the fewest runs decides); then the Wilson lower bound over all your own current runs, pass or fail, earns points, so a small sample earns little.
- Second device (self): every action needs your own passing current run on at least two distinct device fingerprints. Emulator and hardware fingerprints both count, and both are flagged.
- Before / after: a no-skill vs with-skill pair clip (demos/before-after.webm, .mp4 or .gif, 3 MB max) plus demos/before-after.json (schema_version 1, measured seconds, tokens, tool calls and ok for each side). One recorded pair is presentation, never run evidence: it earns docs points only.
- Tier gates (v1.3): Proven needs 35 points, your own passing current run on every action, at least one demo and 3 validator units on every action; it no longer needs an independent run. Hardened needs at least one independent device class on top of its v1.2 gates, and Reference needs Hardened, so the top two tiers still need other people's runs, community usage and maintenance.
- A manifest is valid only when the loader reports no missing required field; official skills get no exception.
- Emulator runs are recorded as such (emulator: true); artifacts without the field are reported as unknown, never as hardware.
- Simulated (fake-device) runs, unsigned, tampered, mis-attributed artifacts and artifacts whose verifier field is missing are rejected and listed under rejected.
- Trust is separate from quality: official (ghost), benchmark (imported AndroidWorld scaffolds), verified (a maintainer recorded a code review in skillhub/REVIEWED), unverified (everyone else). Verified is never automatic.
- Downloads and likes are displayed and worth 0 points. Point values change only at season boundaries with 30 days notice.
- Not enforced yet (planned): reciprocal verification caps, 14-day account age, random re-execution, tier freeze on a failing newer app version.
Maintenance points are worth their full value for 30 days after the last passing run that was not self-recorded, then fall linearly to 0 at 60 days.
A worked example: ghost/block-night
Every number below is read from this skill's entry in registry.json when the site is built, not typed. It is the highest-scoring
skill in the registry today: trust official, tier proven, 41 of 100 points.
| Category | Cap | Earned |
|---|---|---|
| Docs and packaging | 20 | 20 |
| Self-evidence | 30 | 21 |
| Independent verification | 30 | 0 |
| Community usage (90 days) | 10 | 0 |
| Maintenance | 10 | 0 |
| Subtotal | 100 | 41 |
The evidence behind those points is labelled by who ran it. Self-recorded runs: 50/50 passed (the maintainer ran them, not independent). Maintainer-signed replays: 0/0. Independent runs signed by another account: 0/0. It covers 0 independent device cells; 1 cell is recorded, none of which count as coverage until another account runs there. Its maintenance points would decay linearly to 0 over 60 days after the last non-self passing run.
Next rung: Hardened needs 60 points, 19 more than it has and every gate of that tier listed above. Independent and maintainer runs are what the empty categories are waiting for.
Freshness
| Re-verified within 90 days | x1.0 |
| 90 to 180 days | x0.8, marked Stale |
| Over 180 days | x0.5 |
| Failing run on a newer app version than the last pass | Planned (not enforced yet): tier frozen within 24 h until a pass arrives |
Device coverage
A device cell is one combination of four axes: app version × OS version × screen (WxH + density) × locale.
Only verified independent runs on the current skill cover a cell; repeated runs increase its run count, not its coverage count.
Failures count as runs too: coverage means tested, not passed. Maintainer and self runs are recorded as “not independent”.
Missing fingerprint axes appear as * and cover no cell. Skills default to en-US, but a run must record its locale.
The registry publishes quality.runs.device_grid. For one release, independent_devices
is an alias for device_grid.independent_cells, a cell count, not a count of physical devices. This display change does not change the points formula.
Run counts and the confidence floor
A skill page and the leaderboard show the raw count of counted runs, passed / total, split by evidence class:
self-recorded by the author, maintainer-signed replays (a maintainer-held key re-running the author's steps, which is
reproducibility, not independence), and independent runs signed by another account. Runs on an emulator image are flagged.
No percentage is shown as a success rate.
Below the count, a skill page prints a confidence floor: the lower bound of the 95% Wilson score interval over the counted runs, with no prior. It is deliberately conservative: 3 of 3 passed gives a floor of 44%, 32 of 32 gives 89%, so the floor rewards more runs and never exceeds the observed rate.
For sorting only, the leaderboard breaks ties between skills with equal points and tier on a Wilson lower bound computed after adding a prior of 10 passes and 10 failures. That number is never displayed, because read as a rate it would say one run in three fails when none did.
Anti-gaming rules
- Downloads and likes are shown but are worth 0 points.
- Points attach only to signed run artifacts that verify against a published key.
- Self-runs never count as independent. A verification signed by a maintainer-held key is shown as a maintainer-signed replay, not as independent.
- Planned (not enforced yet): reciprocal verification pairs stop counting after the 3rd pair in 30 days.
- One device fingerprint is one report, however many accounts submit it.
- Planned (not enforced yet): accounts younger than 14 days cannot confer verification points.
- Planned (not enforced yet): a random 5% of Hardened and Reference skills are re-executed every month.
- Pages show raw passed/run counts per evidence class. The leaderboard ranks by points, then tier; a Wilson lower bound with a 10 pass / 10 fail prior only breaks ties and is never shown as a rate.
- Tier changes show as provisional for 7 days.
- Point values change only at season boundaries, with 30 days notice.
- Planned (not enforced yet):
generated_with:is required; undisclosed AI generation freezes points for 30 days.
How rep works
Quality points belong to a skill. Rep belongs to a person: your GitHub login, the same name as your author folder and your signed run principal. There are no accounts. Rep rep-v1.0 is built from the same registry, from skillhub/scoring/rep-v1.json.
| Earned by | Rep |
|---|---|
| Published: each skill in your namespace | its current quality points (0 to 100) |
| Verified: independent passing run of someone else's skill | 5 per counted run |
| Device pioneer: first independent pass on a new device class | +3 |
| New bench model: first run with an agent model the author never benched | +3 |
| Maintained: re-verified someone else's skill with a maintainer or platform key | 2 per skill |
| Downloads, likes, stars | 0 |
Badges
| Author | Publishes at least one skill on the hub. |
| Verifier | At least one independent passing run of someone else's skill that counts. |
| Device pioneer | First to verify a skill on a device class nobody had covered for it. |
| Maintainer | Re-verifies other people's skills with a maintainer or platform key. |
- Published: the sum of the current quality points of every skill in your author namespace. Deprecated skills earn nothing. Imported benchmark scaffolds (androidworld) have no human author and earn no rep.
- Verified: points for each independent passing run of the current version of someone else's skill, counted the way the quality ladder counts it: one report per device fingerprint, never on a fingerprint the author recorded on, signed with your own key in SIGNERS.
- Device pioneer bonus: the first independent passing run on a device class the skill had no independent pass on yet.
- Bench model bonus: the first non-author passing run that carries a bench block for an agent model the author never benched this version with. The deterministic replay arm is not a model.
- Maintained: a small fixed amount per skill of someone else's that you re-verified (maintainer or platform key, passing run of the current version). Your own runs of your own skills are self evidence and earn published points only through quality.
- A new version resets run credit: only runs of the current version count, exactly like quality points.
- Downloads, likes and stars are worth 0 rep.
- Not enforced yet (planned): reciprocal verification caps, account age, random re-execution, decay.
Point values live in a versioned scoring file and change only at season boundaries. Past scores are never recomputed under a new formula.