Remove surveys already cleaned from the processing inventory

Compares the current DLW inventory against the PIP master inventory by joining on survey_id and comparing content_hash (DLW) against content_hash_dlw (master). Surveys are kept when they are new to the master (no content_hash_dlw) or when their DLW content hash differs from the previously cleaned value. If the master inventory cannot be loaded, all surveys are returned.

Usage

inv_to_process(inv, verbose = TRUE, dt_master = NULL, master_available = NULL)

Arguments

inv

A data.table of DLW surveys (latest versions).

verbose

Logical. Print progress messages.

dt_master

A data.table of the PIP master inventory, already loaded by the caller (valid_dlw_load()) and shared with the aux-hash comparison. Default NULL, in which case the master is loaded here.

master_available

Logical. Whether the caller already attempted to load the master. When TRUE, dt_master is used as-is. When FALSE, the master was attempted but unavailable, so all surveys are returned without re-loading. Default NULL (unknown — load here if dt_master is NULL).

Value

A data.table of surveys still needing processing, or NULL if all surveys have already been cleaned.

See also

Other pd_process_data pipeline: add_attr(), aux_hash_candidates(), build_pip_inventory(), create_attr(), data_to_dt(), filter_aux_data(), filter_aux_inv(), fix_year_var(), get_aux_hashes(), inv_dlw_load(), log_report(), pd_aux_attr(), pd_deflation(), save_pip_data(), survey_id_to_attr(), valid_dlw_load()