The Role

We're looking for a QA Engineer (Mid or Senior level) who can own quality across AI-driven systems and the integrations that hang off them. This is not traditional app QA. You'll test LLM-powered features, prompt pipelines, agent workflows, MCP integrations, and the GitHub-based delivery pipelines that power them. You'll work directly inside our repos (including degen-engine and skeleton), partner with engineers using Claude Code and Gemini Code Assist, and shape how we verify non-deterministic systems.

If "how do you QA an LLM?" is a question you've already started answering — keep reading.

What You'll Do

- Own end-to-end QA for Skeleton: AI agents, prompt pipelines, MCP server integrations, scheduled jobs (Vercel Cron), data ingestion (Apify), and database flows (Drizzle ORM).
- Design test strategies for non-deterministic systems: evaluation harnesses, golden datasets, regression suites for prompts, output quality scoring, hallucination and drift detection.
- Write and maintain integration tests across our stack (Next.js, TypeScript, pnpm, Vercel, Sentry, Jira) including API contract tests for third-party integrations.
- Test inside GitHub directly: review PRs, run test suites in CI/CD, validate auto-deploys to main, and verify fixes before they ship.
- Partner with engineers using Claude Code, Gemini Code Assist, and our broader AI dev workflow — including writing test prompts, validating tool-use outputs, and stress-testing prompt caching strategies.
- Build and maintain monitoring and observability for AI features in production (Sentry, custom eval dashboards, cost and latency tracking).




- Define quality gates and release criteria for AI-powered features, and partner with engineering on incident response when production outputs drift.
- Triage and reproduce issues across integrated systems — when something breaks, you trace it from Slack notification through Vercel logs, Sentry traces, the database, and back to the prompt.

What We're Looking For

Mid-Level (3–5 years)

- 3+ years of QA / SDET / Test Engineering experience on production software.
- Hands-on experience testing AI / LLM-powered features in production (OpenAI, Anthropic, Gemini, or similar) — prompt evals, output validation, regression testing.
- Strong TypeScript / JavaScript fundamentals; comfortable reading and writing code, not just black-box testing.
- Experience with modern web stacks: Next.js, REST/GraphQL APIs, serverless (Vercel / AWS Lambda), and at least one ORM (Drizzle, Prisma, etc.).
- Fluency in Git and GitHub workflows: PR review, branch protection, CI/CD pipelines, status checks.
- Experience writing automated tests with modern frameworks (Vitest, Jest, Playwright, Cypress).
- Comfort working in repos alongside engineers and contributing test code directly — not just filing tickets.

Senior (5+ years)

- Everything in Mid-Level, plus:



deep experience defining QA strategy for AI / ML systems in production.
- Track record of building eval frameworks for LLM outputs (LLM-as-judge, golden datasets, A/B prompt testing, regression suites for non-deterministic systems).
- Experience with MCP (Model Context Protocol), tool use / function calling, agent frameworks, or multi-step LLM workflows.
- Comfort with observability stacks (Sentry, Datadog, custom dashboards) and ability to build them where they don't exist.
- Experience mentoring engineers on quality practices and shaping team-wide testing culture.
- Familiarity with prompt caching, model selection, context management, and other techniques for keeping AI systems fast and cheap in production.

Nice to Have

- Direct experience with Claude (API, Claude Code, Anthropic SDK), Gemini Code Assist, or similar AI dev tools.
- Experience with Apify, Playwright, or other scraping / browser automation frameworks.
- Background testing data pipelines, ETL flows, or analytics systems.
- Experience with Jira automation, Slack apps, or Notion API.
- Open-source contributions to AI tooling or testing frameworks.
- Curiosity about prompt engineering, agent design, or the science of evaluating language models.

Why You'll Love It Here

- Work on genuinely novel problems: QA for AI systems is being invented right now, and you'll help invent it here.
- Direct access to a small senior team building production AI pipelines from scratch — not a maintenance role, a frontier one.
- Modern stack, modern tools, no legacy debt to drag through.

📌 QA Engineer (Perú)
🏢 Builders - Base
📍 Perú