← All jobs
Mercor Verified
Remote · Hourly

AI Evaluation Rubric Auditor

$60–$85/hr
Share
Posted Nov 16, 2026

The work

We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:

- Was the information available?

- Does the rubric accurately grade what the prompt asked for?

- Would a valid answer have been marked wrong?

- Is the rubric structure correct per the guidelines?

This role includes providing feedback to reviewers regarding prompt and rubric issues.

Must-haves

- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.

- Experience providing feedback to writers/reviewers

- Consistent availability

- Can read a long agent transcript and find where it went wrong.

- Precise writing, and you check the data instead of trusting the reference answer.

Pay range
$60–$85/hr
15h/wk
Share
Earn $340 for each successful referral on this role.
Similar roles

You might also like

Terac Verified New
United States · remote
Accounting Professionals: Interview on Reconciliation Workflows
8h ago
$200 one-time
Terac Verified New
United States · remote
Finance and Operations Leaders: Interview on Reconciliation Infrastructure
14h ago
$300 one-time
Terac Verified New
United Kingdom · remote
Mobile Banking Users: Paid Interview on Banking App Features
15h ago
$10 one-time
Alignerr Verified New
remote · hourly
Public Policy Economist (AI Training)
Yesterday
$35–$70/hr