Software Engineers: Create and Validate AI Coding Tasks

Incentive

Incentives for this project are determined by geography. The following incentive per task applies. Rates are in USD:

• United States: $154
• Canada: $112
• South America: $82
• South Korea: $92
• Philippines: $72
• Each task requires 4 to 5 hours to complete

What We're Researching

We're hiring software engineers for an ongoing remote project focused on building robust evaluations for AI coding agents. Your expertise ensures these models can handle realistic, complex problems in a terminal environment. This work directly feeds into improving the accuracy and reliability of automated engineering tools.

How It Works

You will design realistic, self-contained software-engineering problems from scratch. The work involves implementing features, debugging software, configuring Docker environments, and writing automated tests. You will also create reference solutions and verify that each task can be graded reliably by our systems. All work happens remotely via your own machine using standard command-line tools and GitHub.

Who This Is For

This opportunity is best suited for experienced software engineers who are comfortable navigating unfamiliar codebases. We welcome backend developers, full-stack engineers, DevOps specialists, and systems engineers. Candidates must be highly proficient with command-line tools and writing clean, deterministic tests.

Responsibilities

• Design self-contained software-engineering problems for AI agents to solve.
• Build and configure development environments using Docker.
• Write automated, deterministic tests to reliably grade AI performance.
• Create accurate reference solutions for complex coding challenges.

Requirements

• Professional experience working as a software engineer.
• Comfortable navigating unfamiliar codebases independently.
• Strong proficiency with GitHub and command-line tools.
• Willingness to complete a one-time skills assessment before starting.