Commands

usage: cpplink <command> [options]

commands:
  inspect     load a parquet file and report cardinality and memory
  profile     what the columns can be worth, what a matching pair will
              score, and which pairs of them are the same evidence twice
  levels      check the schema's fuzzy thresholds against the column they
              run on, and propose better ones
  simplify    merge the levels a scored stream cannot tell apart
  explain     show the levels a single pair lands on, and with a model
              the waterfall of bits behind its score
  explain-blocking  price every blocking source without enumerating
  recall      measure what fraction of known pairs blocking reaches,
              and with --why, diagnose the ones it does not
  estimate    learn m, u and lambda and write the model
  completeness  estimate blocking recall with no known pairs at all
  predict     score the candidate pairs and write the predictions above a
              threshold, as one file or as one shard per thread
  cluster     join the predictions into duplicate clusters, from either a
              merged prediction file or a shard directory
  merge-predictions  combine the prediction shards into one csv or parquet file
  rescore     re-score a spilled run under a new model, without comparing again
  gen-sample  write a sample parquet file with planted duplicates

options:
  -h, --help       show this message and exit
  -v, --version    show the version and exit

Every command that reads data takes one or more parquet files. They are read in order into one store and each becomes a dataset, so two files mean linking rather than deduplicating — see linking two files.

What each one is for

Command	Answers	Reads	Writes
inspect	Did the data load as I expected, and what does it cost in memory?	parquet + schema	stdout
profile	What can these columns be worth, what will a matching pair score, and which pairs of them are the same evidence twice?	parquet + schema	stdout
levels	Are my fuzzy thresholds anywhere near right for this column?	parquet + schema	stdout, optionally a schema
simplify	Which levels is this run too small to tell apart, and what does dropping them save?	parquet + schema + model	stdout, optionally a schema
explain	Why did this pair get that pattern, and with --model, that score?	parquet + schema (+ model)	stdout
explain-blocking	How many candidate pairs will this plan cost me?	parquet + schema	stdout
recall	What fraction of true matches does blocking even reach?	parquet + schema + truth	stdout
estimate	What are m, u and λ?	parquet + schema	model.json
completeness	What fraction of true matches does blocking reach, with no truth file?	parquet + schema + model	stdout
predict	Which pairs score above the threshold?	parquet + schema + model	one prediction file, or one shard per thread
rescore	What would a different model have scored?	parquet + schema + model + spill	the same, one file or shards
cluster	Which records are the same entity?	parquet + schema + predictions (a file or a shard directory)	clusters.csv
merge-predictions	Give me the predictions as one file something else can open.	prediction shards (+ parquet + schema for the ids)	one csv or parquet file
gen-sample	Give me realistic data with known answers.	—	parquet + truth csv

The order you actually run them

gen-sample ──▶ inspect ──▶ profile ──▶ levels ──▶ explain-blocking ──▶ recall ──▶┐
   (or your      sanity      is there    are the   is the plan          does the  │
    own data)     check      anything   thresholds affordable?          plan reach│
                             to find?     right?                       the matches?
                                                                                  │
   ┌──────────────────────────────────────────────────────────────────────────────┘
   └▶ estimate ──▶ simplify ──▶ predict ──────▶ cluster
       model.json   which levels   predictions.  clusters.csv
                    are worth      parquet
                    keeping?

simplify is the one diagnostic that has to come after estimate: it compares each level's m and u against its neighbour's, so it needs a model to read them from. Everything it merges changes how γ packs, so a schema it rewrites needs estimate run again against it.

Tuning is a loop back through the middle of that line, not a rerun of it. predict --spill keeps each pair's agreement pattern, and rescore replays it under a new model in a fraction of the time the first pass took — 0.01 s against 2.2 s on the 1M sample, because every string metric has already been paid for.

The first six are diagnostics and cost seconds. They exist because the two decisions that determine the quality of the output — the comparison levels and the blocking plan — are both made before any expensive work runs, and both are hard to reason about without measurement. profile comes first among them because it needs neither: no model, no blocking plan and no known pairs, just the columns. levels is the one that answers the first of those two decisions, and it needs nothing more than profile does.

Every command at a glance

One invocation of each, in the order above, all against the same schema. Each command's page says what its report means.

# Draft a schema from the file: guess what each column is from its name and its
# type, and write a default comparison per column, the derived columns a role
# wants, and the blocking sources a role is strong enough to carry.
# --role overrides a guess and --id names the record id; without --out the
# schema is stdout and the report stderr
cpplink init --out schema.json data.parquet
cpplink init --role notes=text --role reg_no=national_id data.parquet > schema.json

# Report cardinality, null rates and the memory each structure costs
cpplink inspect --schema schema.json data.parquet

# Ledger what the columns can be worth, what a matching pair will score, and
# which pairs of columns are the same evidence twice; no model, no blocking
# plan, no known pairs
cpplink profile --schema schema.json data.parquet

# Check the schema's fuzzy thresholds against the column they run on, from the
# exact u curve of the dictionary self-join and an m curve from anchor pairs
cpplink levels --schema schema.json --out proposed.json data.parquet

# Show which level each comparison assigns to one pair, and the packed pattern
cpplink explain --schema schema.json --pair r17,r19 data.parquet

# Price every blocking source exactly, without enumerating a single pair
cpplink explain-blocking --schema schema.json data.parquet

# Measure what fraction of known duplicate pairs blocking actually reaches
cpplink recall --schema schema.json --truth truth.csv data.parquet

# Learn m, u and lambda and write the model
cpplink estimate --schema schema.json --out model.json data.parquet

# Merge adjacent levels a run of this size cannot tell apart, which narrows the
# packed pattern; --min-gap decides on how much a level is worth rather than on
# whether the difference is significant, which on a large file it always is
cpplink simplify --schema schema.json --model model.json \
                 --min-gap 1.0 --out simpler.json data.parquet

# Score every candidate pair and write the predictions above a threshold. --out
# names a directory to get one shard per thread, or a .csv/.parquet file to get
# one file: threads still write a shard each and the run merges them at the end.
# -v prints the plan first (every source priced, the threshold, where the output
# goes) and a progress line while the pairs are walked, on stderr
cpplink predict --schema schema.json --model model.json \
                --out predictions.parquet --threshold 20 -v data.parquet

# Re-score a spilled run under a new model, without comparing anything again
cpplink predict --schema schema.json --model model.json \
                --out predictions/ --spill spill/ --threshold 20 data.parquet
cpplink rescore --schema schema.json --model tuned.json \
                --spill spill/ --out predictions2/ --threshold 20 data.parquet

# Join those predictions into duplicate clusters, and score the result against
# known pairs; --predictions takes the merged file or the shard directory, and
# the extension of --out picks csv or parquet
cpplink cluster --schema schema.json --predictions predictions.parquet \
                --out clusters.parquet --truth truth.csv data.parquet

# Combine the shards of an existing run into one file something else can open
cpplink merge-predictions --schema schema.json --shards predictions/ \
                          --out predictions.parquet data.parquet

# Write a sample file with realistic cardinalities and planted duplicates
cpplink gen-sample --out sample.parquet --rows 1000000 --truth sample.truth.csv
# ... or split across files for a link fixture: originals to --out, every planted
# duplicate to one of the --out-b files, so every recorded pair crosses them
cpplink gen-sample --out a.parquet --out-b b.parquet --out-b c.parquet --rows 100000 --truth truth.csv

Every command takes more than one parquet file, and two with nothing said means linking them; see Linking two files for the link-mode forms of the same commands.

Conventions shared by every command

• --schema <file.json> is required by everything except gen-sample. It declares columns, comparisons and blocking sources; see the schema reference.
• The parquet file is a positional argument, always last in the examples here.
• Errors go to stderr prefixed with cpplink: or cpplink <command>:, and the exit code is 1. A configuration error — a level applied to a column type it cannot read, a comparison with no trailing else, a γ that overflows 32 bits — is reported at parse time, before a data file is opened.
• Thresholds are given either in bits of match weight (--threshold) or as a posterior probability (--probability). Prefer bits; see choosing a threshold.
• cpplink::Run(args, out, err) takes explicit streams rather than using std::cout, so every output shown in these pages is captured verbatim in the test suite.