How skills are scored

Every skill carries two separate badges. Trust says who published it and how it got into the hub. Quality says whether it works and how well it is documented. This page is the whole quality formula, scoring v1.3: a score from 0 to 100.

auto-login runner: beta. Action state badges declare prerequisites and transitions: “Requires: logged in”, “Sets: logged in”, or “Clears: logged in”. They do not prove a session exists or add scoring points. Deterministic probes check the current device; signed passing runs remain the evidence.

Trust

Official Written and maintained by the Ghost team under skillhub/ghost/. Says who wrote it, not that anyone else verified it; the Reliability card shows the runs. Installs silently.
Verified Community skill merged into the hub repo through a recorded review.
Benchmark Reference skill imported from a benchmark suite (AndroidWorld) by the maintainers. Not reviewed community code; most are scaffolds with no authored steps yet.
Unverified No recorded review: installed from a URL outside the hub repo, or merged without one. The CLI asks before installing.
Quarantined Flagged by the registry. Install is blocked unless forced.

Quality tiers

A skill needs both the points and every hard gate of a tier to hold it. Tiers are a ladder: each one includes the gates below it.

  1. Draft

    0 points
    • skill.yaml validates
    • Static scan is clean
  2. Runs

    10 points
    • README present
    • One signed self-run artifact (fingerprint hash, app version, per-step result)
  3. Proven

    35 points
    • Your own signed passing run of the current version on every action
    • A demo clip of 3 MB or less (webm, mp4 or gif)
    • 3 validator units on every action (custom_python or a parameterised text check = 2, a bare presence check = 1)
  4. Hardened

    60 points
    • One passing run signed by another account, on a device fingerprint the author has not recorded on
    • 2 or more OEM or Android-major combinations
    • Bench on 2 or more models
    • 10 or more community runs with Wilson lower bound (95%) of 0.80 or higher
    • "Known failure modes" section in the docs
    • Re-verified in the last 90 days
  5. Reference

    90 points
    • Hardened for 60 or more continuous days
    • Curator review
    • No open P1 bug older than 14 days
    • Author at Contributor tier or above
    • Capped at about 5% of Hardened skills per category; re-reviewed every 180 days or on a major target-app release

Where points come from

Each category has a cap, and the caps add up to 100, so no single axis can carry a skill up the ladder.

CategoryCapEarned by
Docs and packaging 20 Manifest +3, Readme +3, Known failure modes +4, Demo clip +4, Demo for every action +3, Before / after pair (clip + metrics) +3
Self-evidence 30 First self-run +4, Bench on one model +4, Second device of your own (every action) +4, Validators (per action) +2 each per action, up to +8 (weights: custom python x2, parameterised text x2, default x1), Self-reliability (your own pass rate) up to +10, Wilson lower bound over your own verified runs once 3 exist on every action
Independent verification 30 First independent +12, Second device class +8, Third device class +4, Bench second model +4, Bench third model +2
Community usage (90 days) 10 Wilson z +1.96, Min runs +10, Window days +90
Maintenance 10 Reverified 30d +6, Bug response 7d +2 each, up to +4
Downloads and likes0Shown on the page, never scored.
score = (min(docs, 20) + min(self_evidence, 30) + min(independent, 30) + min(community_usage, 10) + min(maintenance, 10)) x freshness_multiplier max = 20 + 30 + 30 + 10 + 10 = 100 community_usage = round(10 x wilson_lower(passed_90d, runs_90d)) once enough recent runs exist, else 0

Self-reliability. Up to 10 points inside Self-evidence for the pass rate of your own verified runs, pass or fail: the Wilson lower bound (95%) of passed over total, times 10, rounded. It starts counting at 3 recorded self-runs, so a single lucky run earns nothing and a skill that fails often loses the points on its own evidence.

Validators are scored per action. The action with the fewest validator units sets the number: +2 per unit up to +8, where custom python counts x2, parameterised text counts x2, default counts x1. Ten trivial checks on one action cannot stand in for the rest.

Rules in the scoring file

Verbatim from skillhub/scoring/v1.json (v1.3); the registry builder applies exactly these.

Maintenance points are worth their full value for 30 days after the last passing run that was not self-recorded, then fall linearly to 0 at 60 days.

A worked example: ghost/block-night

Every number below is read from this skill's entry in registry.json when the site is built, not typed. It is the highest-scoring skill in the registry today: trust official, tier proven, 41 of 100 points.

CategoryCapEarned
Docs and packaging2020
Self-evidence3021
Independent verification300
Community usage (90 days)100
Maintenance100
Subtotal10041
score = 41 x freshness 1 = 41

The evidence behind those points is labelled by who ran it. Self-recorded runs: 50/50 passed (the maintainer ran them, not independent). Maintainer-signed replays: 0/0. Independent runs signed by another account: 0/0. It covers 0 independent device cells; 1 cell is recorded, none of which count as coverage until another account runs there. Its maintenance points would decay linearly to 0 over 60 days after the last non-self passing run.

Next rung: Hardened needs 60 points, 19 more than it has and every gate of that tier listed above. Independent and maintainer runs are what the empty categories are waiting for.

Freshness

Re-verified within 90 daysx1.0
90 to 180 daysx0.8, marked Stale
Over 180 daysx0.5
Failing run on a newer app version than the last passPlanned (not enforced yet): tier frozen within 24 h until a pass arrives

Device coverage

A device cell is one combination of four axes: app version × OS version × screen (WxH + density) × locale. Only verified independent runs on the current skill cover a cell; repeated runs increase its run count, not its coverage count. Failures count as runs too: coverage means tested, not passed. Maintainer and self runs are recorded as “not independent”. Missing fingerprint axes appear as * and cover no cell. Skills default to en-US, but a run must record its locale.

The registry publishes quality.runs.device_grid. For one release, independent_devices is an alias for device_grid.independent_cells, a cell count, not a count of physical devices. This display change does not change the points formula.

Run counts and the confidence floor

A skill page and the leaderboard show the raw count of counted runs, passed / total, split by evidence class: self-recorded by the author, maintainer-signed replays (a maintainer-held key re-running the author's steps, which is reproducibility, not independence), and independent runs signed by another account. Runs on an emulator image are flagged. No percentage is shown as a success rate.

Below the count, a skill page prints a confidence floor: the lower bound of the 95% Wilson score interval over the counted runs, with no prior. It is deliberately conservative: 3 of 3 passed gives a floor of 44%, 32 of 32 gives 89%, so the floor rewards more runs and never exceeds the observed rate.

For sorting only, the leaderboard breaks ties between skills with equal points and tier on a Wilson lower bound computed after adding a prior of 10 passes and 10 failures. That number is never displayed, because read as a rate it would say one run in three fails when none did.

wilson_lower(p, n) = (p + z²/2n - z·sqrt(p(1-p)/n + z²/4n²)) / (1 + z²/n), z = 1.96 confidence_floor = wilson_lower(passed / total, total) shown on the page sort_tiebreak = wilson_lower((passed + 10) / (total + 20), total + 20) never shown

Anti-gaming rules

How rep works

Quality points belong to a skill. Rep belongs to a person: your GitHub login, the same name as your author folder and your signed run principal. There are no accounts. Rep rep-v1.0 is built from the same registry, from skillhub/scoring/rep-v1.json.

Earned byRep
Published: each skill in your namespaceits current quality points (0 to 100)
Verified: independent passing run of someone else's skill5 per counted run
Device pioneer: first independent pass on a new device class+3
New bench model: first run with an agent model the author never benched+3
Maintained: re-verified someone else's skill with a maintainer or platform key2 per skill
Downloads, likes, stars0

Badges

AuthorPublishes at least one skill on the hub.
VerifierAt least one independent passing run of someone else's skill that counts.
Device pioneerFirst to verify a skill on a device class nobody had covered for it.
MaintainerRe-verifies other people's skills with a maintainer or platform key.

Point values live in a versioned scoring file and change only at season boundaries. Past scores are never recomputed under a new formula.