Apertus Engineer: Evaluations

We are seeking a skilled engineer to join the Apertus evaluation effort. The ideal candidate will build and operate the evaluation codebase and pipelines that inform our training and release decisions, keeping results consistent between training and serving. This role requires strong Python engineering, hands-on LLM evaluation experience, and the ability to work collaboratively in a research-focused environment.

We train open foundation models with hundreds of billions of parameters on thousands of GPUs on one of the largest AI-ready supercomputers in Europe. The team counts more than a dozen full-time engineers working alongside leading researchers from EPFL and ETH Zürich, has released the Apertus 1 and Apertus 1.5 models, and works with over thirty academic collaborators to deliver fully open (open source), responsibly trained, multilingual, multimodal AI models for research and industry.

Apertus is trained and developed on Alps, the Swiss National Supercomputing Centre's (CSCS) supercomputing infrastructure. The role requires someone who is comfortable working in an HPC environment and collaborating with researchers and infrastructure engineers.

The engineer will own the evaluation codebase and pipelines that inform training decisions and releases.

Evaluation infrastructure

Build and maintain the evaluation codebase and pipelines for Apertus models, from checkpoints during training to released models
Make evaluations run quickly and at scale: parallel execution on Alps, efficient use of inference backends, caching, and result tracking
Reduce mismatch between evaluations during training and during serving: consistent tokenisation, chat templates, prompting, and sampling across evaluation harnesses and inference engines
Debug evaluation failures, regressions, and inconsistencies across backends
Benchmark coverage

Integrate and run the evaluations the project cares about. The design of new evaluations is owned by collaborating researchers and engineers; this role makes them run reliably and at scale
Cover image and audio evaluations alongside text within the same pipeline
Integrate new benchmarks as the field evolves, working with our academic collaborators to onboard the benchmarks they create, and validate that metrics and harness implementations are trustworthy
Comparative and third-party evaluation

Evaluate third-party services and other open and closed models against the same benchmark suite, producing directly comparable and reproducible results
Provide evaluation results, reports, and dashboards that support training decisions (data mixtures, ablations) and release decisions
Work closely with the engineers focused on safety, deployment, and community needs, and integrate the evaluations they create into the shared pipeline
Essential

MSc or PhD in Computer Science, Data Science, Artificial Intelligence, Machine Learning, or a related field. Exceptional
BSc candidates with strong engineering experience will also be considered
Strong Python and software engineering skills, including experience building robust data or evaluation pipelines
Experience with LLM evaluation: established harnesses (e.g. lm-evaluation-harness) or custom benchmark tooling
Strong collaboration and communication skills and ability to work across research and engineering teams
Prior hands-on experience in the core domains of this role is required. This can be project or study based experience; formal work experience is preferred
A high degree of flexibility: priorities, tools, and day-to-day tasks shift with training schedules, releases, and a fast-moving field
Experience running evaluations at scale on GPU clusters (Slurm or similar) and with inference engines such as vLLM or SGLang
Familiarity with agentic evaluation and agentic harnesses: tool use, sandboxed execution environments, benchmarks such as SWE-bench or similar
Experience with multimodal (image or audio) model evaluation
Strongly preferred

An eye for statistical rigor: variance across runs, prompt sensitivity, significance of differences between models
Nice to have

Published research in the domains relevant to this role, or familiarity with recently published research on these topics
Experience with LLM-as-judge pipelines and their calibration
Familiarity with benchmark contamination detection and decontamination practices
Experience visualising and communicating evaluation results to research teams
A stimulating academic environment at one of the world's leading technical universities
Access to Alps, one of the largest AI-ready supercomputers in Europe
The opportunity to work alongside and intersect with leading researchers in the field
Collaboration with top researchers and engineers from EPFL, ETH Zürich, CSCS, and other Swiss institutions
Attractive employment conditions and comprehensive benefits, including the ETH Zürich/EPFL pension plans
Flexible working arrangements, including options for remote work
Professional development opportunities, including conference attendance and specialised training
The chance to contribute to open-source projects with global impact
Being part of Switzerland's sovereign AI development, working on technology with national significance
The role can be based either in Lausanne at EPFL or in Zürich at ETH Zürich
We look forward to receiving your online application with the following documents:

CV/Resume
Cover letter explaining your interest and qualifications
Academic transcripts
Contact information for 2-3 references
Links to GitHub repositories or other examples of your programming work (if available)
Further information about the ETH AI Center and the Swiss AI Initiative can be found on our website. Questions regarding the position should be directed to Dr. Imanol Schlag, email ischlag@ethz.ch (no applications).

Please note that we exclusively accept applications submitted through our online application portal. Applications via email or postal services will not be considered.

For recruitment services the GTC of ETH Zurich apply.