No measured figures are published here yet.

This page describes how calibration is done. It gives no figure for accuracy, error or agreement, and no result from any test set, because none has yet been measured on the media your installation will read.

Calibration is done in each installation, on Indian media that its own analysts label during the baseline weeks. A figure measured somewhere else would describe other media, and before an installation’s baseline weeks are complete no detector has passed validation on Indian media. Until a check has passed, the product shows its scores labelled as such, and verdicts on that kind of media stay inconclusive unless a trusted content credential or watermark decides.

It is stated first so that nothing below is read as a result. What will be published, and in what form, is at the end of the page.

What calibration does

A check returns a raw score. Calibration turns it into a probability that means what it says: among the labelled items, the share that turn out to be what the check said matches the probability they were given.

Media Verdict console
  1. 01

    One calibrator per bucket

    A calibrator is fitted for each check, media type and language bucket, and the validation ledger keeps one row for each check, media type, quality and language bucket. The method is isotonic: a curve that never falls, fitted to the labelled items.

  2. 02

    Fitted on one half, tested on the other

    A bucket’s labelled items are split in half by a fixed hash of each item. The calibrator and the check’s operating threshold are chosen on the calibration half; every figure for the ledger is computed on the other half, which the fit never saw.

  3. 03

    On items from the wild

    The calibrator is fitted on at least 100 labelled frames of real posts from that bucket, never on laboratory data alone.

  4. 04

    Validated only when every condition holds

    At least 200 items from the real-media set that are known to be unaltered; at least 20 in every declared kind of footage, such as low light, regional language, dubbed or with a music bed; a false-positive rate, at its upper bound, under the cap in the installation’s settings; a measure of how well the check separates the two kinds of item above its minimum; and the sign-off of an approver. A row that misses any one is evidence only, with the reason recorded.

  5. 05

    Tested again

    Each month on fresh items, and whenever the check’s model changes: a row is keyed on the model’s own weights, so a new model has no row until it passes. A row that exceeds the cap is demoted and stops counting.

  6. 06

    The same shape for every score

    Coordination is calibrated on at least 50 topics chosen at random and labelled by two analysts, at least 15 of them coordinated. Forecasts are calibrated on a held-out period of at least three weeks with at least 100 cases in which the outcome happened, and pass only when the error in their probabilities is under its cap. Stance is calibrated per language, and a language counts only once its test set reaches 300 posts.

Why it is done on Indian media

What a check sees depends on the media it is shown. News clips in India carry channel logos, tickers and split screens; phone videos are re-encoded each time they are forwarded; memes carry text; tracks are dubbed or carry music. A check that behaves well on one kind of footage can behave differently on another, and compressed copies are a known difficulty.

So the reference set is built from the media the installation will read: Hindi and regional-language news clips from publishers’ own verified accounts, phone videos, memes with text, music beds, silence, and copies of the same items re-encoded by messaging apps and platforms. Two analysts label each item, and changes to the set need two analysts and an approver.

Labels enter calibration only from analysts whose labels agree closely enough on at least 50 items they both labelled. Review queues also carry a random sample of items the checks found nothing in, so that labels are not drawn only from flagged items.

The set grows for the life of the installation, and every release of the product is run against it.

The labels "uncalibrated" and "not yet checked"

  • Uncalibrated

    A score with no validated calibrator behind it

  • Not yet checked

    A forecast not yet validated against what happened

Uncalibrated means the number shown is the check’s own score. The probability is left empty rather than estimated, and a media check in that state never counts toward high certainty. Coordination, stance and the account score work the same way: a score and no probability until each passes its own calibration.

Not yet checked is the forecasts’ form of the same statement. Each forecast is scored once its hour has passed, and until the forecaster has passed calibration on those records its figures are the model’s own and are marked so.

Neither label is an error. Both are the product saying how far a number can be relied on.

What will be published

Once measured, these are published here, in the form in which the ledger records them.

  1. 01

    By check

    For each kind of check and media type: whether it is validated, evidence only or demoted, and why.

  2. 02

    By quality and language

    The same, for each quality grade and each language bucket, including the buckets that have not yet passed.

  3. 03

    With the size of the test set

    The number of genuine and of made or altered items in each bucket’s test half, and in each declared kind of footage.

  4. 04

    The measured figures

    For each row: the false-positive rate and its upper bound at the operating threshold, how well the check separates the two kinds of item, and how closely its probabilities match the outcomes.

  5. 05

    When, and on which model

    The date each row was measured, and the model it was measured on.

  6. 06

    For coordination, stance and forecasts

    Whether each is calibrated, on how many labelled topics, posts or forecast cases, and the same figures in their own terms.

Plan the baseline weeks with us: what your analysts will label, and when each check can first be validated.

Request a demo