> For the complete documentation index, see [llms.txt](https://docs.zingg.ai/zingg/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zingg.ai/zingg/running-zingg/create-training-data/verify-blocking.md). # Verify Blocking How to inspect the blocking model, understand its coverage, and fix missed matches caused by blocking The blocking model filters your dataset before any similarity comparisons happen. Records in different blocks are never compared. This is what makes Zingg scalable, but it also means that if the blocking model puts two matching records in different blocks, those records will never be matched and the match will be missed regardless of how well the similarity model is trained. If your results are missing matches you expect to see, or if the match job is slow, start here. `verifyBlocking` tells you what percentage of your known matching pairs are being blocked together correctly. Run this test after configuring Zingg and again after training if you suspect missed matches. ### **What `verifyBlocking` output contains** The `verifyBlocking` phase produces two output directories under `zinggDir/modelId/blocks/timestamp/`:
Output pathWhat it contains
countsRecord count per block. Shows how many records are in each block. Very large blocks indicate poor blocking specificity.
blockSamplesThe top 10% of records associated with each block. Use this to understand which records are being grouped together and whether the groupings make sense.
{% hint style="success" icon="right-long" %} **Read more**: To run the `verifyBlocking` phase and inspect output coverage→ [Verify Blocking](/zingg/running-zingg/create-training-data/verify-blocking.md) {% endhint %} ### If `verifyBlocking` shows low coverage, what to do in order Poor blocking coverage means matching pairs are being placed in different buckets and never compared. Four causes account for most cases:
Skewed or limited training samples The blocking model learns heuristics from your labeled training data. If your training data is skewed, for example, all your labeled pairs are from a single state, city, or category, the blocking model learns blocking rules that are too narrow for the full dataset. **Fix**: Run `findTrainingData` again and label pairs that represent the full distribution of your data. If your dataset has records from 50 states, your training data should include\ pairs from multiple states not just the largest.
Manually added training samples overriding learned rules If you supplement your training data with manually created pairs rather than pairs selected by `findTrainingData`, those pairs may teach the blocking model rules that are too specific to those exact examples. The blocking model needs representative pairs selected by Zingg's own candidate selection logic, not hand-curated ones. **Fix**: Use `findTrainingData` to generate candidate pairs. Add manually created training data only to supplement coverage gaps, and ensure they are diverse enough that the blocking model can generalize from them.
Too few training examples 30 to 50 labeled `match` pairs is a good starting point for Zingg; these pairs are used to learn the blocking as well as the similarity model. But the blocking model may need more examples, especially on large datasets with many field variations, before it learns blocking rules that are generic enough to cover the full problem space. **Fix**: Run `findTrainingData` and `label` for an additional round. Pay particular attention to `match` pairs that cover different field value patterns: different name formats, address abbreviations, and missing field combinations.
Non-differentiating columns in the schema If your field definitions include columns that are the same or nearly the same across a large proportion of your records, for example, a country code that is "US" for 99%\ of records, those columns produce very large blocks. Large blocks slow down matching and indicate that the blocking model is not creating useful partitions. **Fix**: Mark non-differentiating columns as `DONT_USE` in your field definitions. `DONT_USE` fields are excluded from both blocking and similarity; they appear in output but do not\ influence matching.
### TODO 1. Add more labeled training pairs of the type being missed. Focus on pairs that represent the variation patterns missing from your current training set. 2. Review your field match types. Fields marked `DONT_US`E are excluded from blocking as well as similarity. If a field has a strong identity signal - a consistent identifier that appears across matching records - consider changing it from `DONT_USE` to `FUZZY`. 3. Check whether missed pairs share a common characteristic. If all missed pairs have empty values in a key field, empty fields cannot contribute to blocking. Either remove that field from your blocking config or improve data completeness upstream. 4. If standard blocking consistently misses a specific pattern in your data, consider custom blocking functions. {% hint style="success" icon="right-long" %} **Read more**: * [Label Training Pairs](/zingg/running-zingg/create-training-data/label-training-pairs.md) - how to add more training data * [Configure Zingg](/zingg/running-zingg/configure-zingg.md) - changing field match types * [Blocking Strategies: DEFAULT vs WIDER](/zingg/tuning/blocking-strategy.md) - reorder candidate fields in the blocking tree (Enterprise only) {% endhint %} {% hint style="success" icon="right-long" %} Run `verifyBlocking` in two situations: 1. **Before labeling** - to confirm your blocking is covering expected matches before you invest in training data 2. **After training** - if your match results are missing pairs you know should be there {% endhint %} ### Run the `verifyBlocking` phase {% tabs %} {% tab title="Community" %} ### Python ```python options = ClientOptions([ClientOptions.PHASE, "verifyBlocking"]) zingg = Zingg(args, options) zingg.initAndExecute() ``` ### CLI ```bash ./scripts/zingg.sh --phase verifyBlocking --conf config.json ``` {% endtab %} {% tab title="Enterprise" %} ### Python ```python options = ClientOptions([ClientOptions.PHASE, "verifyBlocking"]) zingg = EZingg(args, options) zingg.initAndExecute() ``` ### CLI ```bash ./scripts/zingg.sh --phase verifyBlocking --conf config.json ``` {% endtab %} {% tab title="Enterprise Snowflake" %} ### CLI ```bash ./scripts/zingg.sh --phase verifyBlocking --conf config.json \ --properties-file ``` {% endtab %} {% endtabs %} ### **Reading the output** `verifyBlocking` produces two output directories under `zinggDir/modelId/blocks/timestamp/`: * `counts` - block size distribution * `blockSamples` - sample records from each block Read the output to inspect block coverage: ```python verify_output = spark.read.parquet( f "{zinggDir}/{modelId}/verifyBlocking" ) verify_output.show() ``` ### **What to do if coverage is low** If `verifyBlocking` shows that many known matching pairs are not being blocked together, try these in order: 1. **Add more labelled training pairs** of the type being missed. The blocking model learns from the same training data as the similarity model. 2. **Check your field match types.** Fields set to `DONT_USE` are excluded from blocking. If a key identity field is set to `DONT_USE`, matching pairs that differ on other fields may end up in different blocks. 3. **Try the `WIDER` blocking strategy** (Enterprise only) if large blocks suggest the tree is over-relying on one or two fields → [Blocking Strategies: DEFAULT vs WIDER](/zingg/tuning/blocking-strategy.md) {% hint style="success" icon="right-long" %} **Consider custom blocking functions** for specialised data patterns → [Custom Blocking and Similarity](broken://pages/DWdyf7az3MmhJaVca3k7) **Read more**: * Blocking model concept and how it fits in the pipeline → [Blocking Model](/zingg/zingg-concepts/how-zingg-learns/zingg-models/blocking-model.md) * Custom blocking functions for advanced tuning → [Custom Blocking and Similarity](broken://pages/DWdyf7az3MmhJaVca3k7) {% endhint %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.zingg.ai/zingg/running-zingg/create-training-data/verify-blocking.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.