← All jobs
Mercor Verified
Remote (United States) · Hourly

SWE-Bench Task Auditor

$70–$90/hr
Share
Posted Aug 28, 2026

Evaluate the quality, correctness, and reproducibility of software-engineering benchmark tasks used to train and evaluate a frontier AI lab's models. You'll assess repository-level tasks, reference patches, test harnesses, and grading integrity — and provide clear, rubric-based written feedback.

Basic Qualifications • 3+ years professional software engineering • Real open-source contribution or maintainer experience (merged PRs, committer / maintainer roles) • Strong ability to audit reference patches, test runners, and Docker isolation, and to detect answer leakage / reward hacking • Fluency across common ecosystems (Python and at least one of Java / Go / TypeScript / C++)

Preferred Qualifications • Familiarity with SWE-Bench (Verified) or similar repository benchmarks • Maintainer history on major Python OSS (Django, Flask, scikit-learn, sympy, pytest, etc.) • Prior code-review or task-grading experience

Pay range
$70–$90/hr
40h/wk
Share
Earn $1,440 for each successful referral on this role.
Similar roles

You might also like

Mercor Verified New
United States · remote · task-based
Agency Management System Administrator / Technology Lead – Paid Research Interview
10h ago
$380/hr
Turing Verified New
remote
Domain Expert – Python
Posted Nov 13, 2026
Pay on listing
micro1 Verified New
remote · hourly
Software Engineer (Staff), Financial Platform & Infra
Posted Nov 13, 2026
$7–$8/hr
Alignerr Verified
remote · full-time
Senior Software Engineer — Agentic Coding
Posted Nov 13, 2026
$80–$120/hr