Position: AI Benchmark Quality Reviewer

Type: Contractor Assignment

Compensation: $30/hr

Location: Remote

Commitment: 40 hours/week, with at least 8 hours overlap during PST business hours

Role Responsibilities

- Validate task quality by reviewing instructions, source materials, reference solutions, and evaluation criteria for consistency, completeness, and hidden requirements.
- Review AI agent performance by inspecting execution traces, tool calls, and generated deliverables to determine whether successes and failures are appropriately assessed.
- Audit grading logic by identifying brittle checks, incorrect expected answers, unsupported rubric criteria, and cases where valid alternative solutions are unfairly penalized.
- Investigate discrepancies and distinguish model limitations from task defects, grader errors, environment issues, and tool failures.

Requirements





- 5+ years of relevant professional experience in technical QA, software testing, AI evaluation, data analysis, benchmark development, validation, or a related technical field.
- Comfortable reading Python, SQL, shell scripts, structured data, and execution logs to understand task setup and grading behavior.
- Strong analytical skills with practical experience checking calculations, reconciling conflicting evidence, and assessing the correctness and completeness of technical deliverables.
- Experience providing clear, specific, and reproducible technical feedback.
- Strong written English and attention to detail.

Application Process

- Apply via Easy Apply.
- Check your email for next steps.

📌 AI Quality Analyst | $30/hr Remote (Perú)
🏢 Crossing Hurdles
📍 Perú