Skip to contents

Process DLW inventory and create cleaned pip data

pd_process_data.Rd

Iterate over the datalibweb (DLW) inventory, process each survey by merging auxiliary data (PFW, CPI, PPP, population, GDP, PCE), cleaning main variables, creating metadata, and saving new versions of the cleaned data and metadata into the pip storage. The function returns an updated pip inventory with the new versions recorded.

Usage

pd_process_data(
  inv = NULL,
  aux_measures = c("pfw", "cpi", "ppp", "pop", "gdp", "pce"),
  force = FALSE,
  verbose = getOption("pipdata.verbose", default = TRUE),
  force_surveys = NULL,
  bootstrap = FALSE,
  bootstrap_entities = NULL,
  dependency_plan = NULL
)

Arguments

inv

A data.frame or tibble containing the completed DLW validation inventory. Default NULL, in which case it is loaded internally via pipload::load_gmd_valid_inv(). Before planning or row lookup, input is normalized to data_available = "Yes" rows whose status is "valid" or "invalid". Recognized legacy blank/"No" retry rows are excluded; malformed completed rows abort rather than entering cleaning.

aux_measures

A character vector of auxiliary measures to load and merge with the DLW data. The default is c("pfw", "cpi", "ppp", "pop", "gdp", "pce").

force

Logical. If TRUE, forces reprocessing of all surveys by switching stamp versioning to "timestamp" and bypassing the master inventory comparison. Default FALSE. For surgical re-processing without the global versioning side effect, see force_surveys.

verbose

Logical. Print progress messages. Default: getOption("pipdata.verbose", default = TRUE).

force_surveys

Character vector of survey_id and/or pip_id values to re-process surgically, alongside the normal invalidation candidates. Mutually exclusive with force = TRUE. Preserves content-based stamp versioning (unlike force = TRUE, which switches to timestamp versioning for the entire run). Unknown identifiers are warned about and skipped. Default NULL.

bootstrap

Logical. Explicitly permit rebuilding unknown legacy provenance. Default FALSE.

bootstrap_entities

Optional restrictive survey/pip identifiers for a bootstrap canary. Unlike force_surveys, this never expands selection.

dependency_plan

Optional precomputed advisory plan. Execution validates and restricts it again before any processing side effect.

Value

A data.frame: updated pip inventory (new_pip_inv) with new versions for cleaned data and metadata.

Details

Validation handoff: Both valid and invalid completed validation rows keep their existing cleaning eligibility. Execution-failure control rows are not eligible. The guard is applied here and again during dependency execution so legacy inventories cannot create cleaning, metadata, or deflation actions.

Logging: This function writes process_summary_inf and null_svys_inf entries to the "pipdata_log", summarizing totals and failed surveys. Additional entries for auxiliary file changes and inventory verification are emitted by valid_dlw_load() and build_pip_inventory() respectively.

Aux hashes: the current content_hash for every requested auxiliary measure is resolved once from the "aux" stamp catalog via get_aux_hashes() before aux data is loaded. The run-level hash map is passed to build_pip_inventory() and recorded in the master inventory so that valid_dlw_load() can gate aux-change detection against the aux data actually used in this run.

Recode spec: the recode specification is synced to stamp once via sync_recode_spec() before the per-survey loop and the resolved spec is threaded into each process_data() call, so apply_recode_spec() performs no stamp I/O per survey.

Memory management: surveys are processed one at a time. After each survey is saved, the large intermediates (df, ls_cpfw, ls_clean, metadata) are explicitly removed and gc() is called inside process_data() before the next survey is loaded, keeping peak heap bounded on full-inventory runs.

Examples

if (FALSE) { # \dontrun{
release <- "20250203"
pipfun::setup_working_release(release)
pd_process_data()
} # }