Skip to contents

Save cleaned PIP data or metadata to versioned storage

save_pip_data.Rd

Iterates over a named list of cleaned data.table objects and writes each one to the PIP storage backend via pipload::pip_write(). Errors during individual saves are caught, logged, and returned as NULL so that remaining surveys can continue.

Usage

save_pip_data(
  data,
  alias,
  verbose = getOption("pipdata.verbose", default = TRUE)
)

Arguments

data

A named list of data.table objects to save. Names are used as the id argument to pipload::pip_write().

alias

Character scalar. The storage alias passed to pipload::pip_write() (e.g., "pip" for survey data, "pip_meta" for metadata).

verbose

Logical. Controls verbosity of downstream pipload::pip_write() calls. Default: getOption("pipdata.verbose", default = TRUE).

Value

A named list with one entry per artifact: list(pip_id, success = TRUE) on success or NULL on failure. Version metadata is persisted to the stamp catalog and read back by build_pip_inventory() — it is not returned here.

Details

Artifacts are written largest-first (by object.size()) so that the largest serialisation buffers are allocated while the heap is cleanest. Before writing any artifact whose in-memory size exceeds getOption("pipdata.gc_threshold_bytes", default = 100e6) (default 100 MB), a gc() cycle is triggered to reclaim fragmented memory and reduce the risk of cannot allocate buffer errors from qs2.

See also

Other pd_process_data pipeline: add_attr(), aux_hash_candidates(), build_pip_inventory(), create_attr(), data_to_dt(), filter_aux_data(), filter_aux_inv(), fix_year_var(), get_aux_hashes(), inv_dlw_load(), inv_to_process(), log_report(), pd_aux_attr(), pd_deflation(), survey_id_to_attr(), valid_dlw_load()