← All jobs
Mercor Verified
Remote · Hourly

AI Evaluation Rubric Reviewer

$40–$75/hr
Share
Posted Nov 16, 2026

The work

We are testing AI agents on realistic shopping and delivery tasks. You'll review AI-drafted tasks (prompt, grading rubric) along with model attempts, and decide whether each failure is fair:

- Was the information available?

- Does the rubric accurately grade what the prompt asked for?

- Would a valid answer have been marked wrong?

- Is the rubric structure correct per the guidelines?

Fix the criteria that aren't fair, flag the tasks that can't be saved, and write a short note on each call.

Must-haves

- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

- Can read a long agent transcript and find where it went wrong.

- Precise writing, and you check the data instead of trusting the reference answer.

Pay range
$40–$75/hr
15h/wk
Share
Earn $300 for each successful referral on this role.
Similar roles

You might also like

Terac Verified
remote · hourly
Kinyarwanda Speakers: Audio Transcription for Speech Recognition
Posted Nov 12, 2026
$4/hr
Terac Verified
remote · hourly
Amharic Speakers: Audio Transcription and Segmentation Project
Posted Nov 12, 2026
$4/hr
Alignerr Verified
remote · hourly
E-commerce Data Analyst
Posted Nov 12, 2026
$40–$120/hr
Alignerr Verified
remote · hourly
Map & Location Data Analyst
Posted Nov 12, 2026
$40–$120/hr