Building Reproducible Analytical Pipelines

Master of Data Science, University of Luxembourg - 2026

Author

Bruno Rodrigues

Published

September 15, 2026

Introduction

This is the 2026 edition of the course. If you’re looking for earlier editions, you can find the 2024 and 2025 archives on GitHub.

What’s new in 2026:

  • T at the center: A modern, domain-specific language purpose-built for polyglot reproducible analytical pipelines (R, Python, and Julia).
  • Pipelines as pure function composition: Reframing pipelines through functional programming and Directed Acyclic Graphs (DAGs).
  • Universal environment reproducibility: Using Nix as the foundation for bit-for-bit identical, cross-language software closures.
  • AI-first workflow: Leveraging Large Language Models (LLMs) to construct and review reproducible pipelines and tests.

This course is based on my book Reproducible Polyglot Data Science.

Schedule

  • 2026-10-30 (1.5 hours): Intro (Why T, the reproducibility spectrum) and Nix (environment reproducibility).
  • 2026-11-11 (14:00 - 17:15, MSA 2.400): Building Pipelines with T (syntax, nodes, Arrow IPC serialization, DAG caching).
  • 2026-11-19 (14:00 - 17:15, MSA 4.050): Pipelines as Function Composition (pure functions, immutability, state elimination).
  • 2026-11-20 (10:15 - 11:45): Professional Workflows I: Testing and Test Oracles (unit testing with {testthat} & pytest, golden datasets, pipeline assertions with testcraft).
  • 2026-11-27 (14:00 - 17:15, MSA 3.190): Professional Workflows II: The AI Harness & Collaborative Git (code inspection literacy for LLM traps, authoring AGENTS.md, closed-loop agent execution with t test --json, interactive Git with git add -p).
  • 2026-12-03 (14:00 - 17:15, MNO 1.030): Packaging Code in R and Python for T pipelines.
  • 2026-12-10 (14:00 - 17:15, MSA 3.120): Containerisation with Docker (spatial vs. temporal reproducibility, packaging Nix & T).
  • 2026-12-16 (12:45 - 15:45, MSA 3.100): Continuous Integration with GitHub Actions & Course Project Wrap-up.

Reproducible analytical pipelines?

This course is my take on setting up code that results in some data product. This code has to be reproducible, documented, and production-ready. The term was originally introduced by the UK’s Analysis Function.

The basic idea of a reproducible analytical pipeline (RAP) is to have code that always produces the exact same result when run, whatever that result might be. This is crucial in research and science, but it is equally essential in industry and data-driven decision-making.

A well-documented RAP eliminates technical debt, prevents “it worked on my machine” disasters, and is readily reusable across future projects.

Why T? Focusing on what truly matters

If you have ever tried building a data science pipeline that combines both Python and R, you know the pain:

  • In Python, you battle dependency hell across pip, virtualenv, conda, poetry, and uv, while wrestling with PYTHONPATH resolution and platform-specific wheels.
  • In R, you configure {renv}, ensure compiler toolchains exist to compile C/C++/Fortran packages, and manage package library paths.
  • To connect them, you end up writing fragile bash scripts or Makefiles, exporting ad-hoc CSV files that lose data types, and praying that everyone runs the scripts in the right order.

Students and practitioners end up wasting 80% of their time dealing with language-specific idiosyncrasies rather than learning reproducible architecture.

To break this cycle, this course puts T (tlang) at the center.

p = pipeline {
  -- 1. Load data
  data = read_csv("datasets/country_level_data.csv")

  -- 2. Clean and derive in Python
  derived = pyn(
    command = <{
      derived = data.assign(
        pct_change = data["average_price_m2_nominal_euros"].pct_change() * 100
      )
    }>,
    deserializer = ^ipc,
    serializer = ^ipc
  )

  -- 3. Visualise in R
  plot = rn(
    command = <{
      library(ggplot2)
      plot <- ggplot(derived, aes(x = year, y = pct_change)) +
        geom_line() +
        theme_minimal()
    }>,
    deserializer = ^ipc
  )
}

build_pipeline(p)

The Pedagogical Justification

T is an experimental domain-specific language that I began designing in 2025. While it is experimental, placing it at the center of this course serves a vital pedagogical purpose:

  1. Focus on Reproducibility, Not Language Quirks: T isolates the essential principles of software reproducibility:
    • Pure Functions: Each node is an isolated function with explicit inputs and a single output.
    • Directed Acyclic Graphs (DAGs): Execution order is derived mathematically from data dependencies, not the arbitrary layout of script files or notebook cells.
    • Hermetic Execution: Execution takes place inside an isolated sandbox guaranteed by Nix.
    • Content-Addressed Caching: Only nodes whose inputs or code have changed get rebuilt.
    • Zero-Glue Polyglot Data Exchange: Cross-language data transfer uses standardized formats like Apache Arrow IPC (^ipc), passing DataFrames between Python, R, and Julia without custom serialization code.
  2. Eliminating Fragile Glue: You no longer need to write custom glue scripts, parse command-line arguments, or track intermediate file names.
  3. Designed for the LLM Era: T was created with a minimalist, constraint-based syntax that LLMs can generate and reason about with high precision. You can let an LLM help write the pipeline definitions, while you and the LLM focus on the R or Python data analysis code inside each node.

By abstracting execution into T and environment management into Nix, we remove the accidental complexity of modern data science so you can master the fundamental concepts.

Data products?

In this course, each of you will develop a data product. A data product is anything that requires data as an input and produces value: an analytical report, a reproducible paper, an interactive dashboard, or an API.

The focus of the course is not on the end product itself (which you can choose based on your interests), but on the pipeline that builds that product reliably, reproducibly, and non-interactively.

Machine learning?

Being skilled in machine learning is not enough to become an effective data scientist. In fact, the model is often the easiest piece to replace. What is non-negotiable is knowing:

  • How to write, test, and modularize code;
  • How to manage environments deterministically;
  • How to work within the command line and version control with Git;
  • How to construct automated DAG pipelines;
  • How to package code and deploy with Docker and Continuous Integration.

What actually is reproducibility?

A project is reproducible if it can be rerun by anyone, anywhere, at any point in time, yielding the exact same results with minimal friction.

The requirements of a RAP

For a project to qualify as a RAP, it must meet several strict criteria: - Available, version-controlled source code (tracked with Git and GitHub); - Deterministic dependencies (managed completely with Nix); - Non-interactive execution (run with a single command, without manual steps in GUIs or notebooks); - Tested logic (unit tests validating functions and pipeline data contracts); - Open-source foundation (reproducible operating system closures via Nix and Docker).

The Reproducibility Spectrum

Reproducibility is not binary; it exists along a continuum (Peng 2011):

The reproducibility spectrum from Peng's 2011 paper.
The reproducibility spectrum from Peng’s 2011 paper.

A project can control for: 1. Code only; 2. Code + language versions; 3. Code + language + package libraries; 4. Code + packages + system-level OS dependencies (Nix / Docker); 5. Hardware architecture and bit-for-bit hermetic closures.

In this course, we aim for the highest practical tier of this spectrum.

Large Language Models

LLMs have transformed how software is developed. In this course, we treat LLMs as power tools: they are highly encouraged for drafting functions, generating tests, and structuring pipelines. However, using an LLM requires you to become a rigorous code reviewer. You will learn how to use Git’s interactive staging (git add -p) and automated test suites to verify that AI-generated code meets your exact expectations before committing it.

Prerequisites

  • Basic programming familiarity with R or Python (loops, functions, data structures).
  • Familiarity with the command line interface (CLI).
  • Linux or macOS operating system (Windows users must use WSL2).

Grading

Grading is based on your final course project. During lecture hours you will learn each concept through live demonstrations and hands-on exercises. At home, you will develop your own pipeline around a dataset of your choice.

At the end of the semester, your repository will be evaluated on how easily and reproducibly it builds:

  • Repository and Documentation (5 points): Hosted on GitHub with a clear, comprehensive README.md.
  • Logic, Packaging, and Tests (5 points): Code is modularized into clean functions or packages, and accompanied by automated unit tests.
  • Deterministic Environment (5 points): All dependencies (R, Python, packages, system libraries) are declared and pinned via Nix.
  • Pipeline Execution (5 points): The entire pipeline runs end-to-end via a single command (build_pipeline in T) producing the final output without manual intervention.

Jargon

T

  • T (tlang): A domain-specific language for defining reproducible analytical pipelines across R, Python, and Julia.
  • Node: A single computation step in a pipeline. Nodes act as pure functions: they take explicit inputs and produce a single serialized artifact.
  • rn() / pyn() / jln(): T node constructors for R, Python, and Julia runtimes.
  • Serializer / Deserializer: The mechanism T uses to write and read node outputs (e.g., ^ipc for Apache Arrow, ^csv, ^parquet, ^rds).
  • Pipeline (pipeline { ... }): A declarative block that defines the DAG of nodes and their data dependencies.
  • build_pipeline(): The function that compiles the DAG, checks the cache, and executes stale nodes in dependency order.
  • read_node(): A function used to inspect the cached output of a specific node interactively.

Nix

  • Nixpkgs: The central repository of over 120,000 packages used by Nix.
  • Derivation: A low-level, cryptographic build instruction in Nix.
  • Nix Store (/nix/store): The read-only directory where all isolated packages and build artifacts reside.
  • Hermetic Sandbox: An isolated build environment with no network access or hidden system dependencies.
  • Binary Cache: Pre-built packages identified by cryptographic hash (e.g. Cachix or cache.nixos.org), avoiding redundant local compilation.

R & Python

  • CRAN / PyPI: The respective package registries for R and Python.
  • Arrow IPC (^ipc): An in-memory columnar data format that allows zero-overhead data transfer between R and Python.
  • testthat / pytest: Standard testing frameworks for R and Python code.

Further Reading

License

This course is licensed under the WTFPL.

WTFPL

Peng, Roger D. 2011. “Reproducible Research in Computational Science.” Science 334 (6060): 1226–27.