Head Of Product - Model Evaluation

Applause is raising the bar for digital quality and employee experience. Recognized as a Top Workplace, Applause provides award-winning software testing and UX research solutions to top brands. Our fully managed services leverage a global team and the world's largest independent testing community. We improve digital experiences for global innovators like Google, Microsoft, PayPal, Starbucks, Vodafone, and BMW.

The Role

As Head of Product - Model Evaluation at Applause, you will lead the development of a strategic new AI evaluation platform—and play a foundational role in bringing it to market from the ground up. You will define the product vision, identify target customer personas, and shape the go-to-market strategy, while building the systems that measure, monitor, and improve AI models in production. As the business matures, you will act as the product leader. This position reports directly to the CTO.

This is a rare opportunity to combine 01 product building with enterprise-scale impact. You’ll create a new category for Applause—extending our leadership in digital quality into AI—by developing capabilities such as LLM-as-a-judge systems, human-in-the-loop feedback pipelines, and model observability frameworks. You won’t just refine an existing product; you’ll define what this business becomes and how it wins in the market and then build the team that scales it.

Key Responsibilities

Define the vision, positioning, and roadmap for Applause’s AI evaluation offering

Identify and develop target customer personas (e.g., AI platform teams, ML leaders, product orgs)

Design and execute the go-to-market strategy, including packaging, pricing, and initial sales motions

Partner with sales and marketing to validate demand, refine messaging, and drive early adoption

Translate market feedback into rapid product iteration and differentiation

LLM Evaluation Systems (in partnership with ML/DS)

Partner with data science and ML teams to design closed loop systems: model evaluation insight improvement

Bring LLM-as-a-judge systems into production use for grading, ranking, and preference modeling

Partner with ML teams on iteration strategies (prompting, fine-tuning, data collection)

Ensure evaluation outputs translate into actionable improvements for customers

Human-in-the-Loop Feedback Systems (in partnership with Community Ops)

Leverage Applause’s global testing community to design scalable human evaluation pipelines

Define workflows, annotation schemas, and quality controls to produce gold data sets

Balance quality, cost, and latency to meet customer requirements

Cross-Functional Leadership

Act as the bridge between product, data science, engineering, and go-to-market teams

Translate complex technical capabilities into clear, differentiated product offerings

Drive alignment on priorities, success metrics, and execution plans

Job Requirements and Preferred Skills

Required

5+ years of product management experience, including work on AI/ML-driven or data products

Experience owning or contributing to 01 product development and go-to-market strategy

Demonstrated ability to take a product from concept to paying customers

Strong ability to define customer personas, value propositions, and product positioning

Experience working closely with data science or machine learning teams

Comfortable engaging at a technical level with ML engineers. You can discuss model architectures, evaluation metrics, and API design without needing a translator

Strong understanding of metrics, experimentation, and data quality

Experience with B2B/enterprise products and technical buyer personas

Preferred

Background in data-science, machine learning, or engineering

MBA or equivalent experience

Experience with model evaluation systems, including human and/or automated approaches

Familiarity with LLM-as-a-judge, pairwise ranking, or preference modeling

Understanding of the limitations and tradeoffs in evaluating generative AI systems

Experience at AI evaluation/tooling companies (eg Arize, Langsmith, Galileo, Scale AI, Weights & Biases, or Humanloop) or as a buyer/implementer of such tools at an enterprise

MBA or equivalent experience in high growth, product led organizations

Why Applause?

You'll help build a new business line from the ground up — with the backing of an established company that already has enterprise relationships with the world's biggest brands. Here's what makes this opportunity unique:

Built-in moat: Applause's global testing community gives you a human-in-the-loop evaluation infrastructure that no AI startup can replicate. You're not starting from zero — you're starting from a million evaluators.

CTO sponsorship: This role reports directly to the CTO with a clear mandate and executive air cover to build fast.

Enterprise access: Applause already sells to the Fortune 500. You'll have warm introductions, existing contracts, and credibility that would take a startup years to build.

Category creation: You're not optimizing an existing product. You're defining a new business for a company that's ready to invest in it.

Additional benefits

Flexible work environment with top talent from across the globe.

International team of 450+ passionate, talented co-workers.

Hands-on projects providing exposure to well-known, global brands.

Applause Core Values

As a global employee community, we strive to uphold the following core values, which are critical to business success and how we measure individual and team performance. Do you share our core values?

Be Accountable: You love to take ownership, and hold yourself and others accountable to increase empowerment and success.

Celebrate Authenticity: You love bringing your true self to work and creating genuine and trustful relationships within a diverse environment.

In It Together: You have a team-first mindset and love collaborating with your peers.

Create Value for Our Customers: You love delivering meaningful business impact and being a release partner for all aspects of digital quality.

Crush Your Goals: You always strive for excellence and constantly seek ways to be better, more effective and more efficient.

Accommodations

Applause is a place where everyone belongs and where we believe everyone deserves the exceptional. We continue to celebrate diversity and are committed to creating an inclusive, equitable environment for our employees. If you believe you require a reasonable accommodation und.er any of the legally protected characteristics, please click here to complete an accommodation request. Please note, Applause will only review requests for applications that have been submitted. We will review your qualifications and follow up with you regarding your request if your qualifications meet our current needs.

#LI-OB1