How to use Jev Score: rubrics, scores and confidence

When should I use Score?

Use Score when an answer belongs on an ordered scale, such as issue severity, skill level or contribution. Define what each level means, and the model returns a score along that scale.

Choice selects among options without a ranking. Noul judges whether a condition holds. Score measures a degree rather than just whether something is true.

How do I write a Score request?

Put facts and context in state, choose a model with model, and put your questions in questions. Each scoring question uses type: score, instructions to explain what to rate, and criteria to describe the levels.

In this example, the macaque Naruto triggers the shutter of photographer Slater's unattended camera and takes a selfie. Slater later processes, selects and publishes the photographs. Two questions rate their contributions to the same photograph.

The backticked names subject, human and creative_work in instructions refer to fields in state. Question IDs such as subject_contribution identify the answers but are not used for inference, so write the complete question in instructions.

{
  "state": {
    "scenario": "Naruto is a Celebes crested macaque living in the Tangkoko nature reserve in North Sulawesi, Indonesia. David Slater, a wildlife photographer shooting macaques in the reserve, leaves his camera unattended, and Naruto repeatedly activates the shutter, producing hundreds of images, including a remarkably sharp, grinning self-portrait reminiscent of a human selfie. Slater later processes and publishes the best photographs in a book that names him as the copyright owner. The book states that Naruto took the photographs, with captions like, 'Surely a sign of self-awareness?' Another caption reads, 'Naruto the macaque smiles at itself while pressing the shutter button on a camera.'",
    "subject": "Naruto",
    "human": "Slater",
    "creative_work": "the grinning self-portrait"
  },
  "model": "jev-latest",
  "questions": {
    "subject_contribution": {
      "type": "score",
      "instructions": "How much did `subject` contribute to `creative_work`?",
      "criteria": [
        "None",
        "Minorly",
        "Moderately",
        "Majorly",
        "Completely"
      ]
    },
    "human_contribution": {
      "type": "score",
      "instructions": "How much did `human` contribute to `creative_work`?",
      "criteria": [
        "None",
        "Minorly",
        "Moderately",
        "Majorly",
        "Completely"
      ]
    }
  }
}

How do I define a rubric?

Order criteria from low to high, with at least two levels. Array indices start at 0, so None, Minorly, Moderately, Majorly and Completely correspond to 0–4 in this example.

For practical use, define the scope and describe concrete situations at each level. For example, clarify whether preparation, composition, triggering the shutter and editing count. Keep each question to one dimension and test the levels against known examples.

How do I read the response?

answers returns results under your question IDs. score is the probability-weighted value; legend maps level indices to descriptions; probabilities gives the distribution, summing to 1; confidence summarizes certainty.

score = Σ(i × pᵢ), where i is the zero-based level index and pᵢ its probability. The score can therefore fall between integer levels.

Different distributions can produce the same score. All probability on level 2, or half on level 0 and half on level 4, both average to 2. Read the distribution alongside the score.

What does confidence tell me?

confidence is a statistic from 0 to 1 computed from the full probability distribution. It summarizes certainty in that distribution. Score magnitude and certainty are separate: a low score can have high confidence.

It is neither the largest level probability nor a directly interpretable accuracy rate. The official pages checked do not specify the exact formula. Even confidence of 1 does not guarantee a correct judgment.

How does the monkey-selfie calculation work?

The screenshot places most of Naruto's probability on major or complete contribution, and most of Slater's on minor contribution. Read the full distribution before the scores that summarize it.

Using the displayed whole percentages gives 3.52 for Naruto and 1.20 for Slater; the screenshot reports 3.51 and 1.18. Display rounding may explain the small differences, but the original response is needed to confirm their cause.

Naruto's confidence is 59%; Slater's is 68%. A higher contribution score does not imply higher confidence.

Recorded from the case screenshot, not live model results. Probabilities are displayed as whole percentages.

What are the common misinterpretations?

These questions are evaluated independently, not as a division of one total. Their scores do not need to add up to 4. Dividing 3.51 by 4 normalizes a score; it does not establish an actual contribution share of 87.75%.

Contribution also does not establish creative intent or copyright ownership. Naruto triggering the shutter and Slater providing equipment and later processing help interpret the scores, but this is a reading of the story. The screenshot does not provide the model's explanation.

Before using the judgment in a workflow, define contribution clearly, validate the rubric against known outcomes, and read score, probabilities and confidence together.