TRIPLE::PASS LABS

TallyLock

Lock the plan. Tally every run. See the real score.

When someone shares an AI result, readers ask three things. How many tries did it take? Is the difference just noise? Is it comparable to the other numbers? TallyLock answers all three in one line you can paste next to the result, and anyone can recheck it.

pipx install tallylock
Read the source View on PyPI

Free and open source under Apache-2.0. Version 0.1.0a4 is an alpha for Linux with Python 3.11 to 3.14. You can also install it with uv tool install tallylock.

An example claim line, as tallylock score prints it:

0.742 best of 14 runs; adjusted 0.712 ± 0.011 (95% CI, 12 completed, 2 failed); plan 3f2a9c1d (1 of 3 in repo); ledger 9b1e04aa; tallylock check

  • How many tries?

    Fourteen runs sit behind the best score. A best score from many tries is weaker evidence than the same score from one.

    best of 14 runs

  • Is it just noise?

    The real score is the mean of completed runs, with a 95% interval. Failed runs are counted and shown, never averaged in.

    adjusted 0.712 ± 0.011

  • Is it comparable?

    The plan ID names one fixed setup: the commit, uncommitted changes, command and config files. Change any of it and a new plan starts, and the count shows.

    plan 3f2a9c1d (1 of 3 in repo)

  • Can I check it?

    The ledger ID points at the exact record the numbers came from. Paste the line into tallylock check to recompute it.

    ledger 9b1e04aa

Three commands from eval to claim

Run it inside a git repository. Your eval prints a line like tallylock: score=0.74 so TallyLock can read the score. Commit the .tallylock/ folder when you want others to check your claim.

  1. Run

    Put tallylock run -- in front of your eval command. Every attempt is sealed and written to a ledger in order, including the ones that fail. {seed} is filled with the attempt number.

    $ tallylock run --repeat 5 -- python eval.py --seed {seed}

  2. Score

    See the best score, the real score with its 95% interval, and a claim line to paste into a paper, a README or a post.

    $ tallylock score

  3. Check

    Anyone with the record can recompute every number. The check reports a match, a mismatch with the fields that differ, or not found.

    $ tallylock check "<claim line>"

Compare two results

tallylock compare PLAN_A PLAN_B

Says whether two results used the same setup and whether the gap between them is just noise.

Check a timestamp

tallylock verify RECEIPT --online

Checks the Bitcoin timestamp without running a node. Two public services must agree.

Publish a board

tallylock board registry/ --out site/

Builds a static Receipts Board page from a registry of plans. See our board: four Colony Zero models, every run counted.

Why the mean, not the best

Run the same setup again and you get a different result. Random seeds, sampling and timing all move it. The best of many runs drifts up the more you try, even when nothing improved.

The mean of completed runs does not have that bias. So the claim line shows both, and it says how many runs sit behind them.

Illustration: fourteen runs of one setup. The best run sits above the rest. The mean sits in the middle with a narrow interval. Two failed runs are marked and left out of the mean. best mean, with 95% interval failed runs: counted, not averaged
Illustration, not real data.

Outside clocks, not trust

Each ledger can be stamped by public timestamp services. That gives outside evidence that the ledger already held that many runs at that time. If the services cannot be reached, the run still counts and the stamp is marked pending.

No coins, no tokens. Timestamps only.

  • at once
    RFC 3161 timestampsTwo separate time authorities stamp the record right away.
  • hours later
    Bitcoin proofOpenTimestamps anchors the same record in a Bitcoin block. tallylock refresh finishes pending stamps.
  • any time
    Check it yourselftallylock verify --online checks the block without a node, and saved evidence checks again offline.

What a receipt proves, and what it does not

A receipt is evidence about a record. It is not evidence that the result is true.

It shows

  • The plan, the attempts and their outputs existed by a stated time.
  • None of them were changed later. Editing, removing or reordering a ledger line breaks the check.
  • The score counts every recorded attempt, failures included.

It does not show

  • That the eval ran honestly or computes its score correctly. TallyLock records what the command printed.
  • Which model was used.
  • That no attempts happened outside the tool, in another clone, or before the plan existed.

For reviewers

When a result arrives without its history, paste this into the review, the pull request or the email.

Thank you for sharing the headline number. Could you also share an attempt record for it? That is a list of every time this evaluation was run with this setup, including runs that failed or scored lower, with the score of each. A TallyLock plan and ledger is one way to produce it, and tallylock can check it offline on any machine. Any equivalent record that was started before the first run is fine. If some runs were left out on purpose, please say how many and why.