Package 'ksCompare'

Description

Lossy interop: takes the two source frames stored on a arsenal::comparedf object and re-runs ks_compare() using the same BY keys. Useful for teams migrating from arsenal::comparedf() to ksCompare without rewriting their inputs.

Usage

as_ks_comparison(x, ...)

Arguments

Details

Note that ksCompare's diff is computed from scratch — it does not attempt to translate comparedf's internal diff tables row-by-row.

Value

A ks_comparison object.

Description

Treats a ks_comparison as a pass/fail check. By default a "passing" comparison has zero schema, key, and value differences; callers can relax those expectations to allow a budget of known acceptable diffs.

Usage

ks_assert_clean(
  x,
  max_value_diffs = 0L,
  max_schema_diffs = 0L,
  max_unmatched_rows = 0L
)

ks_pointblank_step(
  agent,
  comparison,
  max_value_diffs = 0L,
  max_schema_diffs = 0L,
  max_unmatched_rows = 0L,
  label = "ksCompare clean"
)

Arguments

Details

Two flavours are provided:

• ks_assert_clean() throws a classed error (ksCompare_assertion_failed) and halts a pipeline when expectations are unmet. Returns the comparison invisibly on success, so it can be dropped into a ⁠|>⁠ chain.

• ks_pointblank_step() returns a pointblank step that wraps ks_assert_clean() for use inside pointblank::create_agent() / pointblank::action_levels() flows.

Value

ks_assert_clean() returns x invisibly when expectations are met. On failure, raises a condition of class ksCompare_assertion_failed (catchable in CI with tryCatch(..., ksCompare_assertion_failed = function(e) ...)). ks_pointblank_step() returns a pointblank step.

Examples

a <- data.frame(id = 1:3, x = 1:3)
ks_compare(a, a, by = "id") |> ks_assert_clean()

# Allow a small budget of known diffs
## Not run: 
  ks_compare(a, b, by = "id") |>
    ks_assert_clean(max_value_diffs = 5)

## End(Not run)

Description

Aggregates the note column of cmp$value_diff into a tidy taxonomy: one row per distinct cause, with counts and the columns it affects. Helps answer "what kinds of discrepancies do we have?" before drilling into individual cells.

Usage

ks_cause_summary(x)

Arguments

Details

Compound notes (multiple cues joined by "; ") are split into their elementary cues, so a single cell can contribute to several causes. Cells with note = NA (a "plain" value change with no recognised cue) are bucketed as "plain value change".

Value

A tibble with columns:

• cause: short label (e.g. "letter case differs").

• n_cells: number of cells contributing this cue.

• n_columns: number of distinct base columns affected.

• columns: comma-separated sample of the affected columns (capped at 5 names). Sorted by n_cells descending. Zero-row tibble when there are no value differences.

See Also

ks_compare(), ks_row_diff_summary().

Examples

a <- data.frame(id = 1:3, x = c("a", "b ", "c"), y = c(1, 2, 3))
b <- data.frame(id = 1:3, x = c("A", "b",  "c"), y = c(1, 2, 4))
cmp <- ks_compare(a, b, by = "id")
ks_cause_summary(cmp)

Description

Builds an options object consumed by ks_compare() that controls missing-value semantics, label / format comparison, string normalisation, and time-zone handling.

Usage

ks_comp_options(
  na_equal = getOption("ksCompare.na_equal", TRUE),
  sas_special_missing = getOption("ksCompare.sas_special_missing", TRUE),
  compare_labels = getOption("ksCompare.compare_labels", TRUE),
  compare_formats = getOption("ksCompare.compare_formats", TRUE),
  str_trim = getOption("ksCompare.str_trim", FALSE),
  str_case = getOption("ksCompare.str_case", "sensitive"),
  str_norm = getOption("ksCompare.str_norm", "none"),
  tz = getOption("ksCompare.tz", "preserve"),
  path = getOption("ksCompare.path", NULL)
)

Arguments

Value

A ks_comp_options S3 list.

Global defaults via options()

Each argument falls back to a global R option in the ⁠ksCompare.*⁠ namespace, then to the package default. Precedence is: explicit argument > getOption("ksCompare.<arg>") > built-in default. That makes it possible to set project-wide defaults once and then call ks_compare() without repeating yourself:

# In .Rprofile or at the top of a script
options(ksCompare.path     = "out/qc",
        ksCompare.str_case = "fold")

ks_compare(base, comp) |> ks_report_html()   # honours the globals

ks_set_comp_options() is a small wrapper around base::options() that takes the same argument names without the ksCompare. prefix.

Recognised options: ksCompare.na_equal, ksCompare.sas_special_missing, ksCompare.compare_labels, ksCompare.compare_formats, ksCompare.str_trim, ksCompare.str_case, ksCompare.str_norm, ksCompare.tz, ksCompare.path.

See Also

ks_set_comp_options() for setting the ⁠ksCompare.*⁠ globals from R code.

Examples

# Defaults: strict NAs, label & format comparison on
ks_comp_options()

# Loose strings: trim padding and ignore case
ks_comp_options(str_trim = TRUE, str_case = "fold")

# Suppress label/format drift, but keep cell-level strictness
ks_comp_options(compare_labels = FALSE, compare_formats = FALSE)

# Treat any NA as a difference (PROC COMPARE without NOMISS)
ks_comp_options(na_equal = FALSE)

# Pin every report from this run to a dedicated folder
ks_comp_options(path = tempfile("ksCompare_"))

# Pick up a project-wide global path
withr::with_options(
  list(ksCompare.path = tempfile("ksCompare_")),
  ks_comp_options()$path
)

Description

ks_compare() is the single entry point for the package. It compares base against comp and returns a ks_comparison S3 object containing schema, key, row, and value differences plus a small QC manifest.

Usage

ks_compare(
  base,
  comp,
  by = NULL,
  mapping = NULL,
  tolerance = ks_tol(),
  coerce = c("safe", "strict", "lossy"),
  dup_keys = c("first", "last", "keep_all", "all_pairs", "error"),
  options = ks_comp_options(),
  base_name = NULL,
  comp_name = NULL,
  find_patterns = FALSE,
  n_first_last = 5L,
  max_unmatched_rows = 100L,
  loglevel = c("verdict", "verbose")
)

Arguments

Value

A ks_comparison object with components:

• meta: counts, keys, matching strategy, row-key lookup table.

• schema_diff: one row per matched / unmatched column with kind / label / format comparison.

• key_diff: counts of matched / base-only / comp-only rows.

• row_diff: long table of key_id, base_row, comp_row, status.

• value_diff: long table of differing cells with column_base, column_comp, kind, base, comp, diff, note.

• pattern_summary: detected recurring shapes per column (only populated when find_patterns = TRUE; otherwise an empty tibble with the standard column layout).

• unmatched_rows: full base-only / comp-only rows (capped by max_unmatched_rows); see ks_unmatched_rows().

• first_last_unequal: per-column first / last n_first_last differing observations in key_id order (PROC COMPARE-style).

• verdict: a list with headline (character), severity ("ok", "info", "warn", or "critical"), pct_match (0–100 numeric), n_cells, and n_diffs; the same summary shown by print().

• options, tolerance: the inputs (echoed for the manifest).

• manifest: input hashes + run metadata.

Inspect with print(), summary(), tibble::as_tibble(), or render via ks_report_html() / ks_report_xlsx().

Pipeline

1. Schema alignment — columns are paired by an explicit mapping first, then by exact name match. Unmatched columns become base_only / comp_only and are reported in cmp$schema_diff.

2. Key resolution — by is taken literally if supplied, auto-inferred from shared columns when by = "auto", or rows are matched by row position when by = NULL.

3. Row matching — keyed-unique join, duplicate-key resolution (dup_keys), or position-zip; details land in cmp$meta$matching (also displayed by print() and in the HTML report).

4. Cell diff — per-column equality respecting ks_tol() (abs / rel / ULP) for numerics, ks_comp_options() for strings / NAs / SAS special missings, and vctrs::vec_cast() for compatible types.

5. Pattern detection — recurring shapes across diffs (constant offset, sign flip, trim-only, …) are scored and surfaced on cmp$pattern_summary.

6. Manifest — xxhash64 digests of inputs / options / tolerance plus a timestamp and package version, for QC traceability.

Examples

a <- data.frame(id = 1:3, x = c(1, 2, 3), y = c("a", "b", "c"))
b <- data.frame(id = 1:3, x = c(1, 2, 4), y = c("a", "B", "c"))

# Strict compare on `id`
cmp <- ks_compare(a, b, by = "id")
cmp
tibble::as_tibble(cmp)

# Tolerant numeric compare
ks_compare(
  data.frame(id = 1, x = 1.0),
  data.frame(id = 1, x = 1.0001),
  by = "id",
  tolerance = ks_tol(abs = 1e-3)
)

# Renamed key column
ks_compare(
  data.frame(USUBJID = "S001", x = 1),
  data.frame(SUBJID  = "S001", x = 1),
  by = c(USUBJID = "SUBJID")
)

Description

Returns a one-row tibble with the same headline counts as summary.ks_comparison(). Convenient for binding many comparisons together in a QC loop.

Usage

ks_glance(x, ...)

Arguments

Value

A one-row tibble with columns n_base_rows, n_comp_rows, n_matched_rows, n_base_only_rows, n_comp_only_rows, n_matched_columns, n_value_diffs, n_columns_with_diffs.

Examples

cmp <- ks_compare(
  data.frame(id = 1:2, x = c(1, 2)),
  data.frame(id = 1:2, x = c(1, 3)),
  by = "id"
)
ks_glance(cmp)

Description

Computes a small set of metrics that flag suspicious matching: high diff density, many fully-different rows, duplicate keys resolved positionally, and similar patterns that often indicate the wrong key was used.

Usage

ks_match_health(x)

Arguments

Value

A list with components:

• diff_density: n_value_diffs / (n_matched_rows * n_matched_columns) (0-1).

• n_fully_diff_rows: matched rows where every matched column differs (or, more permissively, >= 80% of columns differ).

• pct_fully_diff_rows: as proportion of matched rows.

• n_value_to_na, n_na_to_value: structural NA transitions.

• dup_keys: TRUE if either side had duplicate key values.

• dup_positional: TRUE if duplicates were paired positionally (keep_all) — the most error-prone path.

• row_count_delta: n_comp_rows - n_base_rows (signed).

• flags: character vector of human-readable warnings (possibly empty).

• severity: "ok", "info", "warn", or "critical".

See Also

ks_recommendations(), ks_compare().

Examples

a <- data.frame(id = c(1, 1, 2), x = c(1, 2, 3))
b <- data.frame(id = c(1, 1, 2), x = c(9, 9, 3))
cmp <- ks_compare(a, b, by = "id", dup_keys = "keep_all")
ks_match_health(cmp)

Description

Synthesises ks_match_health() flags, ks_suggest_key() hints, and a few other heuristics into a short list of user-facing "what to look at first" recommendations.

Usage

ks_recommendations(x)

Arguments

Value

A tibble with one row per recommendation:

• severity ("critical", "warn", "info", "ok")

• title short label

• message longer human-readable text

• action suggested next step (may be NA)

See Also

ks_match_health(), ks_suggest_key().

Description

Produces a polished, single-file HTML report summarising a ks_comparison. The output uses htmltools for layout and reactable for interactive tables. No internet access, no Quarto, and no Pandoc are required.

Usage

ks_report_html(
  x,
  path = NULL,
  title = "ksCompare report",
  subtitle = NULL,
  max_rows = 100L,
  theme = c("default", "slate"),
  group_by_key = FALSE,
  max_groups = 200L
)

Arguments

Details

Layout: a left-hand sticky table of contents, a header with dataset names, a status pill, KPI cards, then sections for schema, row diff, value diff, patterns, and the run manifest. Tables use friendly column labels, column groups, and tick/cross markers instead of raw TRUE/FALSE.

Value

Invisibly returns path (or the assembled tag list when path = NULL).

Examples

## Not run: 
  cmp <- ks_compare(iris, iris, by = "Species")

  # Explicit path
  ks_report_html(cmp, "report.html", title = "ADSL QC")

  # Auto-generated filename in working directory
  ks_report_html(cmp)

  # In-memory tagList for embedding
  tags <- ks_report_html(cmp, path = NA)

## End(Not run)

Description

Writes a multi-sheet workbook summarising a ks_comparison. Sheets: Summary, Schema, KeyDiff, Values, Patterns, OUT_BASE, OUT_COMP, OUT_DIF, OUT_NOEQUAL, Manifest. Cells with value differences are highlighted on the wide OUT_DIF sheet.

Usage

ks_report_xlsx(x, path = NULL, highlight = TRUE, threshold = 0)

Arguments

Value

Invisibly returns path.

Examples

## Not run: 
  cmp <- ks_compare(iris, iris, by = "Species")
  ks_report_xlsx(cmp, "report.xlsx")

  # Auto-named output in a pinned folder
  cmp2 <- ks_compare(
    iris, iris, by = "Species",
    options = ks_comp_options(path = tempfile("ksCompare_"))
  )
  ks_report_xlsx(cmp2)

## End(Not run)

Description

Tallies the number of cell differences per matched observation, sorted descending. Useful for spotting rows where most/all columns disagree (often a sign of a wrong row match, a duplicated key resolved differently, or a systemic shift on a single record).

Usage

ks_row_diff_summary(x, n = NULL)

Arguments

Details

Only matched rows are reported; unmatched rows are available via ks_unmatched_rows().

Value

A tibble with one row per matched observation that has at least one cell diff:

• key_id, base_row, comp_row, key_label

• n_diffs: count of differing cells on this observation.

• columns: comma-separated sample of the affected columns (capped at 8 names). Sorted by n_diffs descending. Zero-row tibble when there are no value differences.

See Also

ks_compare(), ks_cause_summary().

Examples

a <- data.frame(id = 1:3, x = 1:3, y = 1:3)
b <- data.frame(id = 1:3, x = c(1, 9, 9), y = c(1, 9, 3))
cmp <- ks_compare(a, b, by = "id")
ks_row_diff_summary(cmp)

Description

Thin wrapper around base::options() for the ⁠ksCompare.*⁠ namespace. Accepts the same argument names as ks_comp_options() (without the ksCompare. prefix) and forwards them into base::options(), returning the previous values invisibly so the call can be unwound with base::options() or withr::with_options().

Usage

ks_set_comp_options(...)

Arguments

Details

Globals set this way become the new defaults of ks_comp_options() (and therefore of ks_compare() when no ⁠options =⁠ argument is supplied).

Value

Invisibly, a named list of the previous values (suitable for do.call(options, prev) to restore).

Examples

old <- ks_set_comp_options(path = tempfile("ksCompare_"), str_case = "fold")
ks_comp_options()$path
do.call(options, old)  # restore

Description

Scans matched columns to find candidates whose combination with the current by key would make the join key unique on at least one side (and ideally both). Returns columns ranked by how much they reduce duplicate-key cardinality. Useful when ks_compare() used dup_keys = "keep_all" and the result looks suspicious.

Usage

ks_suggest_key(x, base = NULL, comp = NULL, top_n = 10L)

Arguments

Details

Because ks_compare() does not snapshot the input data, the original base and comp frames must be passed in to inspect candidate columns. When base / comp are not supplied (or do not contain the original columns) an empty tibble is returned.

Value

A tibble with columns column, pct_unique_base, pct_unique_comp, would_make_unique. Zero rows when no key is in use, no duplicates exist, or no improvement is possible.

See Also

ks_match_health(), ks_compare().

Description

SAS's ⁠&SYSINFO⁠ macro variable encodes the result of ⁠PROC COMPARE⁠ as a bitmask. ks_sysinfo() returns a compatible-style integer summarizing the same kinds of facts, plus a named integer vector showing which bits are set. Bit positions follow SAS documentation:

Usage

ks_sysinfo(x)

Arguments

Details

Length and informat are not currently tracked; their bits are always clear. haven does not surface SAS informats, and column "length" is not a meaningful R-side concept for numerics.

Value

An integer with class ks_sysinfo. Use as.integer() for the raw value or print to see the named bits.

Examples

a <- data.frame(id = 1:3, x = c(1, 2, 3))
b <- data.frame(id = 1:3, x = c(1, 2, 4), z = 1:3)
cmp <- ks_compare(a, b, by = "id")
ks_sysinfo(cmp)
as.integer(ks_sysinfo(cmp))

Description

Returns the long-format cell-level diff table. Equivalent to as_tibble(cmp) and identical to cmp$value_diff when include_unmatched = FALSE.

Usage

ks_tidy(x, ...)

## S3 method for class 'ks_comparison'
ks_tidy(x, include_unmatched = FALSE, ...)

Arguments

Details

When include_unmatched = TRUE, rows describing observations present on only one side of the comparison are appended to the result with kind = "base_only" / "comp_only" and column_base / column_comp set to NA. One row is added per unmatched observation, capped at ks_compare(max_unmatched_rows = ...).

Value

A tibble of value differences, optionally with appended unmatched-row markers. Has columns key_id, base_row, comp_row, column_base, column_comp, kind, base, comp, diff, na_flow, note.

Examples

cmp <- ks_compare(
  data.frame(id = 1:2, x = c(1, 2)),
  data.frame(id = 1:3, x = c(1, 3, 4)),
  by = "id"
)
ks_tidy(cmp)
ks_tidy(cmp, include_unmatched = TRUE)

Description

Builds a tolerance object consumed by ks_compare(). Two numeric values a and b are considered equal when any of the active rules pass:

Usage

ks_tol(abs = 0, rel = 0, ulp = 0, per_column = NULL)

Arguments

Details

• abs(a - b) <= abs

• abs(a - b) <= rel * max(abs(a), abs(b))

• ulp_distance(a, b) <= ulp — IEEE-754 units in the last place, useful for catching floating-point round-trip noise without false positives on genuine differences.

Per-column overrides may be supplied via per_column, a named list whose names are base-side column names and whose values are themselves the result of ks_tol(). Columns not listed fall back to the top-level abs / rel / ulp.

Value

A ks_tol S3 list.

Examples

ks_tol(abs = 1e-9)
ks_tol(rel = 1e-6, per_column = list(price = ks_tol(abs = 0.005)))
ks_tol(ulp = 4)

Description

Returns the rows that are present on only one side of a comparison ("base_only" or "comp_only"), with their original data columns. Mirrors the "Observations in only" tables produced by SAS ⁠PROC COMPARE⁠.

Usage

ks_unmatched_rows(x, side = c("both", "base_only", "comp_only"))

Arguments

Details

The result is precomputed at ks_compare() time and capped by the max_unmatched_rows argument (default 100). When the cap kicks in, the truncated attribute is set and the full counts are recorded on n_total.

Value

A tibble with columns side, key_id, key_label, base_row, comp_row, followed by the data columns from the side that holds each row (base_row is NA for comp_only rows and vice versa, making it trivial to look an observation up in the original base / comp frame). Empty (zero-row) tibble when there are no unmatched rows.

See Also

ks_compare(), ks_tidy().

Examples

a <- data.frame(id = 1:3, x = c(1, 2, 3))
b <- data.frame(id = 2:4, x = c(2, 3, 4))
cmp <- ks_compare(a, b, by = "id")
ks_unmatched_rows(cmp)

Description

These helpers reshape the long-format value_diff table inside a ks_comparison into the four classic SAS PROC COMPARE output datasets:

Usage

as_outbase(x)

as_outcomp(x)

as_outdif(x)

as_outnoequal(x)

Arguments

Details

• as_outbase() — values from the base frame, only for matched rows that contain at least one differing column; one row per matched key, one column per compared variable.

• as_outcomp() — same shape as as_outbase() but values from the compare frame.

• as_outdif() — numeric difference (base - comp) for every matched row that contains at least one differing column; non-numeric cells are reported as NA with a .note column.

• as_outnoequal() — only the rows where at least one matched cell differs, in long format (one row per differing cell).

The shapes mirror SAS's ⁠OUTBASE=⁠, ⁠OUTCOMP=⁠, ⁠OUTDIF=⁠, and ⁠OUTNOEQUAL=⁠ datasets in spirit; we do not replicate the exact metadata columns (⁠_TYPE_⁠, ⁠_OBS_⁠, ...) but include key_id and the original key columns so the rows can be related back to the source frames.

Value

A tibble.

Examples

a <- data.frame(id = 1:3, x = c(1, 2, 3), y = c("a", "b", "c"))
b <- data.frame(id = 1:3, x = c(1, 2, 4), y = c("a", "B", "c"))
cmp <- ks_compare(a, b, by = "id")
as_outbase(cmp)
as_outcomp(cmp)
as_outdif(cmp)
as_outnoequal(cmp)

Description

Computes a small list of headline counts that drive the printed summary, the HTML report KPI cards, and CI gates.

Usage

## S3 method for class 'ks_comparison'
summary(object, ...)

Arguments

Value

A ks_comparison_summary list (also pretty-printed via print()) with components:

• n_base_rows, n_comp_rows: input row counts.

• n_matched_rows, n_base_only_rows, n_comp_only_rows: row counts after matching.

• n_matched_columns, n_base_only_columns, n_comp_only_columns: schema-side counts.

• n_value_diffs: number of differing cells in matched rows.

• n_columns_with_diffs: how many distinct columns hold those differences.

See Also

ks_glance() for a tibble version, ks_tidy() for the long cell-level table.

Examples

cmp <- ks_compare(
  data.frame(id = 1:3, x = c(1, 2, 3)),
  data.frame(id = 1:3, x = c(1, 2, 4)),
  by = "id"
)
summary(cmp)