3  Pipelines as Function Composition

3.1 Learning Outcomes

By the end of this chapter, you will:

  • Understand the fundamental problem of mutable state in scripts and computational notebooks.
  • Grasp the mathematical and practical definition of pure functions (determinism and zero side effects).
  • See why every T node is a pure function, and why a pipeline is function composition.
  • Write pure helper functions in R and Python that plug into pipeline nodes.
  • Understand how external scripts (script = ...) keep domain logic orchestrator-agnostic.

3.2 Introduction: The Hidden Problem with State

In the previous chapter, we introduced T and saw how to declare polyglot pipelines with nodes and serialization contracts. Now we take a step back and ask: why is T designed this way? Why mandatory pipelines? Why must each node produce exactly one artifact? Why does the declaration order inside pipeline { } not matter?

The answer lies in a body of programming principles called functional programming (FP). You do not need to become a lambda calculus theorist to build reproducible pipelines. But understanding the core ideas—purity, composition, and immutability—will make you a vastly better data scientist, a better author of R and Python code, and a more effective collaborator with AI assistants.

Consider a typical data analysis script:

# Step 1
data <- read.csv("sales.csv")

# Step 2
data <- data[data$revenue > 0, ]

# Step 3
data$margin <- (data$revenue - data$cost) / data$revenue

# Step 4
model <- lm(margin ~ region, data = data)
# Python equivalent
import pandas as pd
data = pd.read_csv("sales.csv")
data = data[data["revenue"] > 0]
data["margin"] = (data["revenue"] - data["cost"]) / data["revenue"]

This code runs. But it possesses a hidden and dangerous property: state. Each step mutates the variable data in the global environment. The script can only be understood by reading it from top to bottom, in exact sequence.

If scripting with mutable state is a crack in the foundation of reproducibility, then computational notebooks (like Jupyter) are a gaping sinkhole. Notebooks introduce an even more insidious form of state: cell execution order. You can run cells out of order, re-run cell 3 after cell 7, or delete a cell whose variable persists in memory. What you see on screen no longer reflects the computational state of the kernel!

This is the root problem that functional programming solves.

3.3 Pure Functions: The Mathematical Ideal

In mathematics, a function has a beautiful property: for a given input, it always returns the exact same output. The function \(f(x) = x^2\) does not depend on what you computed earlier, nor does it mutate anything outside itself.

A pure function in programming satisfies two conditions:

  1. Deterministic: Given the same inputs, it always produces the same output. It depends only on its explicit parameters—never on global variables or hidden state.
  2. Zero Side Effects: It does not alter anything outside its scope. It does not modify global variables, rewrite files unexpectedly, or mutate input objects in place.

Compare these two functions:

# Impure: secretly depends on a global variable
threshold <- 0

filter_positive <- function(df) {
  df[df$value > threshold, ]  # What is threshold? Who defined it? When?
}

# Pure: all inputs are explicit
filter_above <- function(df, threshold) {
  df[df$value > threshold, ]  # Everything required is explicitly passed
}

The pure version is predictable. You can test it in isolation, run it in parallel, and reason about it without needing to know what happened elsewhere in the program.

3.4 T Nodes Are Pure Functions

This is not a metaphor. A T pipeline node is, by construction, a pure function:

  1. Explicit Inputs: Upstream data is passed into the node’s sandbox as named inputs. There is no global environment to accidentally read from.
  2. Explicit Single Output: A node returns exactly one thing: the artifact written to the Nix store by its serializer. A node cannot secretly alter another node’s output.
  3. Hermetic Execution: Because Nix runs the node inside a hermetic sandbox with no outside network or filesystem leakage, the execution is guaranteed to be pure.

Look at a concrete example:

p = pipeline {
  -- Node 1: Produce data. Pure: inputs = none, output = DataFrame (^ipc)
  data_node = rn(
    command = <{
      data.frame(
        x = c(1.0, 2.0, 3.0, 4.0, 5.0),
        y = c(0.0, 0.0, 1.0, 1.0, 1.0)
      )
    }>,
    serializer = ^ipc
  )

  -- Node 2: Fit model. Pure: inputs = data_node (^ipc), output = model (^pmml)
  model_node = rn(
    command = <{
      data_node$y <- as.factor(data_node$y)
      glm(y ~ x, data = data_node, family = binomial(link = "logit"))
    }>,
    deserializer = ^ipc,
    serializer = ^pmml
  )
}

build_pipeline(p)

data_node takes no inputs and returns a DataFrame. model_node takes one DataFrame (data_node) and produces a model artifact. Neither can reach outside its sandbox.

Because nodes are pure functions, T can cache them safely: if data_node has not changed, T never re-evaluates it. Furthermore, the declaration order inside pipeline { } does not matter. T infers the execution graph solely from data dependencies.

3.5 A Pipeline Is Function Composition

When you write a pipeline in T, you are composing pure functions:

\[\text{Pipeline} = (f_n \circ \dots \circ f_2 \circ f_1)(\text{data})\]

This is identical to the pipe operator in R (|>) or method chaining in pandas:

# Function composition in R
mtcars |>
  filter(am == 1) |>
  select(mpg)

T elevates this concept to the architectural level. Each step is an explicit, named, cached pure function that can run in any language.

-- Pure R node
raw = rn(command = <{ read.csv("data.csv") }>, serializer = ^ipc)

-- Pure Python node composing on R output
transformed = pyn(
  command = <{
    transformed = raw.assign(log_val = numpy.log(raw["val"]))
  }>,
  deserializer = ^ipc,
  serializer = ^ipc
)

Because both nodes are pure, the serializer/deserializer contract (^ipc) functions as the type signature between them. The underlying runtime is an implementation detail!

3.6 Writing Clean, Pure Helper Functions

The pure function model applies not just to T nodes as a whole, but to the R and Python code you write inside those nodes.

To keep your code reproducible: - Never mutate inputs in place: Return a new object instead of modifying an existing one. - Never rely on global variables: Pass all configuration parameters explicitly. - Isolate file I/O: Separate functions that calculate values from functions that read/write disks.

3.6.1 Decoupling Logic: External Scripts

You do not have to write all your code inline inside <{ ... }>. In fact, professional projects keep domain logic in plain .R or .py files using the script parameter:

p = pipeline {
  cleaned = rn(
    script = "src/clean.R",
    functions = ["src/utils.R"],
    serializer = ^ipc
  )

  model = pyn(
    script = "src/train.py",
    deserializer = ^ipc,
    serializer = ^pmml
  )
}

Notice the tremendous advantage of this approach: src/clean.R and src/train.py carry zero T-specific syntax. They are standard, idiomatic R and Python scripts. They can be: - Sourced and debugged interactively in your IDE. - Tested using standard tools (pytest, testthat). - Reused by colleagues who do not use T.

T acts purely as the orchestration layer: handling scheduling, caching, sandboxing, and data exchange.

3.7 Summary

  • Mutable state in scripts and notebooks creates fragile, order-dependent code.
  • Pure functions are deterministic and side-effect free.
  • T nodes are pure functions running in isolated Nix sandboxes.
  • A pipeline is a Directed Acyclic Graph of function compositions.
  • Writing your analytical code in plain, pure R and Python files keeps your work testable and portable.