Staff Applied Scientist - Dashboards

New Yorkfull timeOn Site

senior

H1B: yes

Datadog Staff Applied Scientist at Datadog leads the evaluation strategy for Dashboards AI system, ensuring quality across multiple metrics and data sources. This role builds evaluation datasets and regression harnesses to catch quality changes before release.

Responsibilities

Own the evaluation strategy for Dashboards, as well as sister teams within our organization. Define the metrics — offline and online, quality and cost, single-turn and multi-turn — that the team and the broader organization optimize against.
Build the eval datasets, golden traces, and regression harnesses that catch quality changes before they hit customers, and make those assets reusable by every team that is building dashboards and widgets through agents
Drive measurable improvements to retrieval relevance, tool-selection accuracy, and context efficiency, partnering closely with the engineers on the team.
Provide technical leadership across the Dashboards team and the broader organization through design reviews, working groups, and mentorship.

Qualification

Significant experience with evaluation

Required

You have a BS/MS/PhD in a scientific field, or equivalent experience.
10+ years of relevant engineering or applied science experience, including time as a technical lead.
Proven track record of leading ML or GenAI initiatives in a product-driven environment, from research through production.
Significant experience with evaluation, experimentation, or measurement of ML systems at scale.
You bring a strong product mindset and are comfortable driving initiatives across cross-functional teams.
You thrive in ambiguity and can make sound technical calls when the path isn’t yet defined.
Datadog values people from all walks of life. We understand not everyone will meet all the above qualifications on day one. That s okay. If you’re passionate about technology and want to grow your skills, we encourage you to apply.

Benefits

Dental
401(k)
PTO
Equity/RSU
Annual Bonus
Wellness

Company

Datadog

View company page

5001-10000 employees

Observability

Founded 2010

Source: greenhouse

Active within the past hour

More roles at Datadog

Applied AI Architect, Enterprise Tech

San Francisco, CA

Applied AI Architect, Commercial

New York City, NY

Anthropic Fellows Program, The Anthropic Institute (Economics & Policy)

British Columbia, CA

Anthropic Fellows Program, Reinforcement Learning

British Columbia, CA