The new PIP Auxiliary Data Pipeline

The engine that keeps PIP’s auxiliary data reproducible, versioned, stable and auditable

PIP Technical Team

2026-06-01

Introducing the New PIP Pipeline

An interconnected ecosystem of R packages — each covering a distinct pipeline stage, all sharing the same guarantees: release-scoped, versioned, validated, logged, and tracks changes.

Release-scoped by design

Every artifact is tied to a working release — nothing floats outside a versioned context.

{pipfun}

Smart, dependency-aware rebuilding

Steps are never reprocessed without a reason — changes propagate only where needed.

{pipfun} {stamp}

Versioning & time-travel

Content-hashed artifacts with full lineage tracking — any version, any point in time.

{stamp}

Audit logging & error resilience

Every step is logged — successes, warnings, failures. The pipeline never fails silently.

{pipfun}

Double-stage validation

Input and output checks at every stage — errors caught early, never propagated silently.

{pipdata} {pipaux}

Cross-release comparison

Structured diffs across releases and vintages — structural, type-level, and value-level.

{myrror}

{pipfun} Release coordination & logging PIP

{stamp} Versioning & lineage General purpose

{pipload} Data, Metadata & Inventories loading PIP

{pipdata} Survey data preparation PIP

{pipaux} Auxiliary data orchestration PIP

{myrror} Comparison & diff reporting General purpose

Auxiliary Data - The Big Picture

What is the aux data pipeline responsible for? How is it built?

One main orchestrator - {pipaux}

An R package that manages the 20+ auxiliary measures that feed the World Bank’s global poverty estimates. For example, price data, national accounts, demographics and poverty measurement parameters.

Takes raw data from different sources, cleans, validates, and formats each one into ready-to-use datasets for PIP’s poverty estimates.

Responsibility 1

Update & Synchronization Pipeline

Raw → validated → versioned artifacts on the Y-drive — dependency-ordered, rebuilt only when something changed.

Responsibility 2

Comparison Engine

Structured diff reports across releases and within-release vintages.

Aux Data Pipeline Objectives

1

Consistent raw data management
GitHub as a reliable, release-linked staging area for all raw inputs

2

Automated processing for every measure
Pipelines triggered only when needed — avoiding redundant rebuilds while never missing a required update

3

Dependency-aware execution
A dependency graph that resolves order and propagates changes automatically

4

Double-stage data validation
Validation checks at both input and output stages for every measure

5

Release-scoped outputs
Every output file is explicitly tied to a specific production release

6

Versioning and traceability
Every version of every dataset is saved with full metadata — always accessible, always traceable

7

Full audit logging & error resilience
Every step is logged — successes, failures, and errors — so the pipeline never fails silently

8

Cross-release comparison
Tools that surface what changed, where, and when — across releases and vintages

How do we get there?

The new aux data pipeline at a glance - high level overview of its structure, components and workflow

High level Structure

Dependency Graph

Rebuild

Live Pipeline Log

Thank you for your attention!

Questions, comments?

Rossana Tatulli · PIP Technical Team · June 2026