.json`.
```json
{
"fieldDefinition": [
{
"fieldName": "id",
"matchType": "dont_use",
"dataType": "string",
"primaryKey": true
},
{
"fieldName": "job_title",
"matchType": "fuzzy",
"dataType": "string",
"postProcessors": "STANDARDISE_jobtitles"
}
]
}
```
# User Defined Mapping Match Types
Defining match types to match better and faster
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
**Leveraging domain expertise to push matching accuracy. Also save time massaging data before matching.**
While the Zingg matching algorithms learn the variations from the user labels, completely different attributes pose a challenge to matching accuracy. In the case of nicknames, or company abberviations, domain input can greatly enhance matching accuracy. Zingg's `MAPPING` match type is built for such cases.
To leverage user expertise for nicknames for example, the user can supply Zingg with a mapping json file `nicknames.json` which denotes nicknames. Zingg has prebuilt mappings for company names, nicknames etc. Or you can define your own mappings too.
Here is the structure of the json:
[
["Will", "Bill", "William"],
["John", "Johnny", "Jack"],
["Robert", "Rob", "Bob", "Bobby"],
["Charles", "Charlie", "Chuck"],
["James", "Jim", "Jimmy"],
["Thomas", "Tom", "Tommy"]
...
]
Each line here represents common nicknames which represent the same name.
To use this mapping within Zingg, define the field's match type as `MAPPING_` which in our case would be `mapping_nicknames`
```json
"fieldDefinition":[
{
"fieldName" : "name",
"matchType" : "mapping_nicknames, fuzzy",
"fields" : "name",
"dataType": "string"
}
```
### Transform and Match
The `MAPPING` match type can also be used to transform and normalise categorical data. Let us say different data sources have different representations of gender. In one, gender is represented as M and F, in another it is noted as Male, Female and in the third as 1 and 2. Instead of transforming the gender column beforehand, one could create a mapping json called `gender.json` .
[
["M", "Male", "1"],
["F", "Female", "2"]
]
When we use it for the gender field like below, Zingg would automatically handle the transformation so that you don't have to.
```
"fieldDefinition":[
{
"fieldName" : "gender",
"matchType" : "mapping_gender, exact",
"fields" : "gender",
"dataType": "string"
},
```
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Deterministic Matching
Ensuring higher matching accuracy and performance
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
Zingg Enterprise allows the ability to plug rule-based [deterministic matching](https://www.learningfromdata.zingg.ai/p/deterministic-matching-vs-probabilistic) along with already Zingg AI's probabilistic matching. If the data contains *sure* identifiers like emails, SSNs, passport-ids etc, we can use these attributes to resolve records.
The deterministic matching flow is weaved into Zingg's flow to ensure that each record which has a match finds one, probabilistically, deterministically or both. If the data has known identifiers, Zingg Enterprise's Deterministic Matching highly improves both matching accuracy and performance.
### Example For Configuring In JSON:
```json
"deterministicMatching":[
{
"matchCondition":[{"fieldName":"fname"},{"fieldName":"stNo"},{"fieldName":"add1"}]
},
{
"matchCondition":[{"fieldName":"fname"},{"fieldName":"dob"},{"fieldName":"ssn"}]
},
{
"matchCondition":[{"fieldName":"fname"},{"fieldName":"email"}]
}
]
```
#### Python Code Example:
```{python}
detMatchNameAdd = DeterministicMatching('fname','stNo','add1')
detMatchNameDobSsn = DeterministicMatching('fname','dob','ssn')
detMatchNameEmail = DeterministicMatching('fname','email')
args.setDeterministicMatchingCondition(detMatchNameAdd,detMatchNameDobSsn,detMatchNameEmail)
```
#### How Will It Work:
The above conditions would translate into the following:
1. Those rows which have **exactly** same `fname`, `stNo` and `add1` => exact match with max score 1\
\&#xNAN;*OR*
2. Those rows which have **exactly** same `fname`, `dob` and `ssn` => exact match with max score 1\
\&#xNAN;*OR*
3. Those rows which have **exactly** same `fname` and `email` => exact match with max score 1
[^1]: Zingg Enterprise is an advance version of Zingg Community with production grade features
# Pass Through Data
Supressing records entirely during matching yet preserving them in the output
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
In many cases, our input consists for records which are partially populated, outdated or not fit for matching in some way or other. One likely cause for such data is a source system with poor quality information. For governance and compliance purposes, we still want these records in the identity graph, but we do not want them to play a role in matching.
Pass Through Configuration allows such records to appear in the Zingg output but be ignored during matching, linking and incremental processes. Each pass through record gets a unique ZINGG\_ID
### Example For Configuring In JSON:
```json
"passthroughExpr": "is_deceased = true"
```
Note: Zingg internally applies the **negation** of the `passthroughExpr` to filter matching records. If the passthrough condition is being applied on nullable fields, please ensure that the negative of the passthrough expression yields the records which are NOT passthrough. For example:
```json
"passthroughExpr": "is_deceased = true AND is_deceased is NOT NULL"
```
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Model Location
#### zinggDir
The **location** where trained models will be saved. Defaults to `/tmp/zingg`
#### modelId
An **identifier** for the model. You can train multiple models - say, one for **customers** matching *names*, *age*, and other personal details and one for **households** matching *addresses*. Each model gets saved under `zinggDir/modelId`
# Tuning Label, Match And Link Jobs
#### numPartitions
The number of Spark partitions over which the input data is distributed. Keep it equal to 20-30 times the number of cores. This is an important configuration for performance.
#### labelDataSampleSize
Fraction of the data to be used for training the models. Adjust it between 0.0001 and 0.1 to keep the sample size small enough so that it finds enough edge cases fast. If the size is bigger, the `findTrainingData` job will spend more time combing through samples. If the size is too small, Zingg may not find the right edge cases.
# Telemetry
Product usage data captured by Zingg OSS to improve user experience
#### collectMetrics
Application captures a few measurements for runtime metrics such as *no. of data records, no. of features, running phase,* and a few more.
**Zingg does not capture any user data or input data and will never do so.**
This feature may be disabled by setting this flag to **false**. The default value is **true**. For details, refer to [Security And Privacy](/latest/security).
# Working With Training Data
Zingg builds models to predict *similarity*. Training data is needed to build these models. The next sections describe how you can use the **Zingg Interactive Labeler** to create the training data.
# Finding Records For Training Set Creation
Pairs of records that match or dont so as to train Zingg
The **findTrainingData** phase prompts Zingg to search for edge cases in the data which can be labeled by the user and used for learning. During this phase, Zingg combs through the data samples and judiciously selects limited representative pairs that can be labelled by the user. Zingg is very frugal about the training so that user effort is minimized and models can be built and deployed quickly.
This **findTrainingData** job writes the edge cases to the folder configured through `zinggDir/modelId` in the config:
`./zingg.sh --phase findTrainingData --conf config.json`
The **findTrainingData** phase is run first and then the label phase is run and this cycle is repeated so that the Zingg models get smarter from user feedback.
# Labeling Records
Providing user feedback on the training pairs
The label phase opens an *interactive* learner where the user can mark the pairs found by **findTrainingData** phase as matches or non-matches. The **findTrainingData** phase generates edge cases for labeling and the label phase helps the user mark them.
`./zingg.sh --phase label --conf config.json `
Proceed running **findTrainingData** followed by label phases till you have at least 30-40 positives, and when you see the predictions by Zingg converging with the output you want. At each stage, the user will get different variations of attributes across the records. Zingg performs pretty well with even a small number of training, as the samples to be labeled are chosen by the algorithm itself.
**How much training is enough**
The labelled data is the real configuration for Zingg, through which the different Zingg models get created. Heuristically, 30-40 matching records are a good start before you train and match. Actual numbers vary on the size of data, number of attributes, cluster sizes and performance needs. More training, like going to 500-1000 matches, will surely help with performance on almost any dataset. Point to remember here is - Is the effort worth it :smile:
The **showConcise** flag when passed to the Zingg command line only shows fields which are **NOT DONT\_USE**
# Find And Label
This phase is composed of two phases namely [findTrainingData](/latest/stepbystep/createtrainingdata/findtrainingdata) and [label](/latest/stepbystep/createtrainingdata/label). This will help experienced users to quicken the process of creating training data.
`./zingg.sh --phase findAndLabel --conf config.json`
As this is phase runs **findTrainingData** and **label** together, it should be run only for small datasets where **findTrainingData** takes a short time to run, else the the user will have to wait long for the console to label.
# Using Pre-existing Training Data
Supplementing Zingg With Existing Training Data
If you already have some training data that you want to start with, you can use that as well with Zingg. Add an attribute **trainingSamples** to the config and define the training pairs.
The training data supplied to Zingg should have a **z\_cluster** column that groups the records together. The **z\_cluster** uniquely identifies the group. We also need to add the **z\_isMatch** column which is **1** if the pairs *match* or **0** if they do *not* match. The **z\_isMatch** value has to be the same for all the records in the **z\_cluster** group. They either match with each other or they don't.
An example is provided in [GitHub training data](https://github.com/zinggAI/zingg/tree/0.7.0/examples/febrl/training.csv).\
Here, the first column specifies the z\_cluster, the second column specifies the z\_isMatch value and the remaining columns are the ones which are used for training the model.
The above training data can be specified using [trainingSamples attribute in the configuration.](https://github.com/zinggAI/zingg/tree/0.7.0/examples/febrl/configWithTrainingSamples.json)
**Note**: It is advisable to still run [findTrainingData](/latest/stepbystep/createtrainingdata/findtrainingdata) and [label](/latest/stepbystep/createtrainingdata/label) a few rounds to tune Zingg with the supplied training data as well as patterns it needs to learn independently.
# Updating Labeled Pairs
N**ote: Please keep a backup copy of your model folder in a separate place before running this**
As our understanding of our data changes, we may need to revisit the previously marked pairs and update them. To do this, please [generate the documentation of the model.](/latest/stepbystep/createtrainingdata/generatingdocumentation)
You can then invoke the updater by invoking\
`./scripts/zingg.sh --phase updateLabel --conf `
This brings up the console labeler which accepts the **cluster id** of the pairs you want to update.
# Documenting The Training Data
We may need to send our labeled data for a subject matter expert to review. We may also like to understand and analyse our training data. In such scenarios, we can document the records we have marked during the **label** phase by invoking the **generateDocs** phase.
During **generateDocs**, Zingg generates readable documentation about the training data, including those marked as matches, as well as non-matches. The documentation is written to the **zinggDir/modelId** folder and can be built using the following
```
./scripts/zingg.sh --phase generateDocs --conf
```
The generated documentation file can be viewed in a browser and looks like as below:
# Model Difference
Comparison of two outputs with different models
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
## Understand Your Model Changes with Precision
When you evolve your Zingg model—changing field definitions from fuzzy to exact, adding new match types, introducing deterministic matching rules, or tuning blocking strategies—you need to understand the impact. The **diff** phase provides a detailed comparison between two model outputs, showing you exactly which records and clusters were affected by your changes. Make data-driven decisions about which model performs better for your use case.
## Why Use Diff?
* **Model Comparison**: Understand the impact of changing field types (fuzzy → exact, adding new match types)
* **Validation & Testing**: Verify that model improvements actually produce better results
* **Impact Analysis**: See which specific records and clusters changed between model versions
* **Confidence Building**: Make informed decisions about deploying new models to production
* **A/B Testing**: Compare different model configurations to find the optimal setup
* **Regression Detection**: Ensure new model changes don't degrade match quality
* **Cluster Evolution**: Track how clusters merge, split, or change as your model evolves
## How It Works
The diff phase compares two configurations:
1. **Original Configuration**: Your baseline model (e.g., with fuzzy matching)
2. **New Configuration**: Your updated model (e.g., with exact matching or new rules)
The diff phase analyzes both outputs and identifies:
* **Records that moved between clusters** due to model changes
* **New clusters** created in the updated model
* **Merged or split clusters** between the two models
## Usage
### Command Line
```bash
./scripts/zingg.sh --phase diff \
--conf examples/febrl/sparkIncremental/configdiff.json \
--compareTo examples/febrl/configBaseline.json \
--properties-file config/zingg.conf
```
**Key Parameters:**
* `--conf`: Points to your **new/updated configuration** (the model you want to evaluate)
* `--compareTo`: Points to your **original/baseline configuration** (the model you're comparing against)
* `--properties-file`: Zingg properties file (optional)
### Python API
Python users can pass configuration objects directly instead of JSON files. See the complete example in `diffExample.py` below.
```python
from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import *
from zinggEC.enterprise.common.TransformedOutputArguments import *
from zinggEC.enterprise.common.EClientOptions import *
# Step 1: Define your ORIGINAL/BASELINE configuration
# This is the model you're comparing against (e.g., with fuzzy matching)
originalArgs = EArguments()
id = EFieldDefinition("id", "string", MatchType.DONT_USE)
id.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1", "string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
originalArgs.setFieldDefinition([id, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn])
originalArgs.setModelId("100")
originalArgs.setZinggDir("./models")
originalArgs.setNumPartitions(4)
schema = "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string"
originalInputPipe = ECsvPipe("originalData", "examples/febrl/test.csv", schema)
originalArgs.setData(originalInputPipe)
originalOutputPipe = ECsvPipe("originalOutput", "/tmp/zinggOutputOriginal")
originalOutputPipe.setHeader("true")
originalArgs.setOutput(originalOutputPipe)
# IMPORTANT: trainMatch phase must already be completed for the original configuration
# The diff phase reads the output from the original model
# Step 2: Define your NEW/UPDATED configuration
# Example: Changed some fields from FUZZY to EXACT, added deterministic matching
newArgs = EArguments()
id = EFieldDefinition("id", "string", MatchType.DONT_USE)
id.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.EXACT) # Changed from FUZZY
add1 = EFieldDefinition("add1", "string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.EXACT) # Changed from FUZZY
areacode = EFieldDefinition("areacode", "string", MatchType.EXACT) # Changed from FUZZY
state = EFieldDefinition("state", "string", MatchType.EXACT) # Changed from FUZZY
dob = EFieldDefinition("dob", "string", MatchType.EXACT) # Changed from FUZZY
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
newArgs.setFieldDefinition([id, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn])
newArgs.setModelId("999")
newArgs.setZinggDir("./models")
newArgs.setNumPartitions(4)
# Added deterministic matching rules
dm1 = DeterministicMatching('fname', 'stNo', 'add1')
dm2 = DeterministicMatching('ssn')
newArgs.setDeterministicMatchingCondition(dm1, dm2)
newInputPipe = ECsvPipe("newData", "examples/febrl/test.csv", schema)
newArgs.setData(newInputPipe)
newOutputPipe = ECsvPipe("newOutput", "/tmp/zinggOutputNew")
newOutputPipe.setHeader("true")
newArgs.setOutput(newOutputPipe)
# IMPORTANT: Run trainMatch on the new configuration first
options = EClientOptions([EClientOptions.PHASE, "trainMatch"])
zinggNew = EZingg(newArgs, options)
zinggNew.initAndExecute()
# Step 3: Create TransformedOutputArguments for diff
diffArgs = TransformedOutputArguments()
diffArgs.setParentArgs(newArgs) # New configuration
diffArgs.setCompareToArgs(originalArgs) # Original configuration to compare against
# Set output path for diff results
diffOutputPipe = ECsvPipe("diffOutput", "/tmp/zinggDiff")
diffOutputPipe.setHeader("true")
diffArgs.setTransformedOutputPath(diffOutputPipe)
# Step 4: Execute diff phase
diffOptions = EClientOptions([EClientOptions.PHASE, "diff"])
zinggDiff = EZingg(diffArgs, diffOptions)
zinggDiff.initAndExecute()
```
## Configuration Files
### Configuration Wrapper (`examples/febrl/sparkIncremental/configdiff.json`)
This wrapper configuration points to your new configuration and specifies where to write the diff output:
```json
{
"config": "$ZINGG_ENT_REPO$/spark/examples/febrl/configdiffUpdated.json",
"transformedOutputPath": {
"name": "diffOutput",
"format": "csv",
"props": {
"location": "/tmp/zinggTransformedOutputDiff",
"delimiter": ",",
"header": true
}
}
}
```
**Note**:
* This wrapper only points to the new configuration. The original/baseline configuration is specified via the `--compareTo` command-line parameter.
* The `name` field in `transformedOutputPath` can be any arbitrary identifier for the output pipe - it's used internally by Zingg to identify this output destination. Here we use `diffOutput` to clearly distinguish it from the new model's match output.
### New Configuration (`examples/febrl/configdiffUpdated.json`)
Your updated model configuration with changes—for example, switching some fields from `fuzzy` to `exact` matching:
```json
{
"fieldDefinition": [
{
"fieldName": "id",
"matchType": "dont_use",
"fields": "id",
"dataType": "string",
"primaryKey": true
},
{
"fieldName": "fname",
"matchType": "fuzzy",
"fields": "fname",
"dataType": "string"
},
{
"fieldName": "lname",
"matchType": "fuzzy",
"fields": "lname",
"dataType": "string"
},
{
"fieldName": "stNo",
"matchType": "exact",
"fields": "stNo",
"dataType": "string"
},
{
"fieldName": "add1",
"matchType": "fuzzy",
"fields": "add1",
"dataType": "string"
},
{
"fieldName": "city",
"matchType": "exact",
"fields": "city",
"dataType": "string"
},
{
"fieldName": "areacode",
"matchType": "exact",
"fields": "areacode",
"dataType": "string"
},
{
"fieldName": "state",
"matchType": "exact",
"fields": "state",
"dataType": "string"
},
{
"fieldName": "dob",
"matchType": "exact",
"fields": "dob",
"dataType": "string"
}
],
"output": [{
"name": "OUTPUT_FEBRL",
"format": "csv",
"props": {
"location": "/tmp/zinggOutputNew/",
"delimiter": ",",
"header": true
}
}],
"data": [{
"name": "test",
"format": "csv",
"props": {
"location": "examples/febrl/testmodeldiff.csv",
"delimiter": ",",
"header": false
},
"schema": "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, state string, areacode string, dob string, ssn string"
}],
"deterministicMatching": [
{
"matchCondition": [
{"fieldName": "fname"},
{"fieldName": "stNo"},
{"fieldName": "add1"}
]
},
{
"matchCondition": [
{"fieldName": "ssn"}
]
}
],
"outputStats": {
"name": "stats",
"format": "csv",
"props": {
"location": "/tmp/zinggStats_$ZINGG_DYNAMIC_STAT_NAME",
"delimiter": ",",
"header": true
}
},
"labelDataSampleSize": 0.5,
"numPartitions": 4,
"modelId": 999,
"zinggDir": "models"
}
```
### Original Configuration (`examples/febrl/configBaseline.json`)
Your baseline model configuration (the one you're comparing against):
```json
{
"fieldDefinition": [
{
"fieldName": "id",
"matchType": "dont_use",
"fields": "id",
"dataType": "string",
"primaryKey": true
},
{
"fieldName": "fname",
"matchType": "fuzzy",
"fields": "fname",
"dataType": "string"
},
{
"fieldName": "lname",
"matchType": "fuzzy",
"fields": "lname",
"dataType": "string"
},
{
"fieldName": "stNo",
"matchType": "fuzzy",
"fields": "stNo",
"dataType": "string"
},
{
"fieldName": "add1",
"matchType": "fuzzy",
"fields": "add1",
"dataType": "string"
},
{
"fieldName": "add2",
"matchType": "fuzzy",
"fields": "add2",
"dataType": "string"
},
{
"fieldName": "city",
"matchType": "fuzzy",
"fields": "city",
"dataType": "string"
},
{
"fieldName": "areacode",
"matchType": "fuzzy",
"fields": "areacode",
"dataType": "string"
},
{
"fieldName": "state",
"matchType": "fuzzy",
"fields": "state",
"dataType": "string"
},
{
"fieldName": "dob",
"matchType": "fuzzy",
"fields": "dob",
"dataType": "string"
},
{
"fieldName": "ssn",
"matchType": "fuzzy",
"fields": "ssn",
"dataType": "string"
}
],
"output": [{
"name": "output",
"format": "csv",
"props": {
"location": "/tmp/zinggOutputOriginal",
"delimiter": ",",
"header": true
}
}],
"data": [{
"name": "test",
"format": "csv",
"props": {
"location": "examples/febrl/testmodeldiff.csv",
"delimiter": ",",
"header": false
},
"schema": "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, state string, areacode string, dob string, ssn string"
}],
"outputStats": {
"name": "stats",
"format": "csv",
"props": {
"location": "/tmp/zinggStats_$ZINGG_DYNAMIC_STAT_NAME",
"delimiter": ",",
"header": true
}
},
"labelDataSampleSize": 0.5,
"numPartitions": 4,
"modelId": 100,
"zinggDir": "models"
}
```
## Output Format
The diff output contains **only the records that were impacted** by the model changes, making it easy to focus on what actually changed.
### Output Fields
The diff output includes:
* **Primary Keys of records** from both configurations
* **ZINGG\_ID\_UPDATED**: ZINGG ID from the new/updated model
* **ZINGG\_ID\_ORIGINAL**: ZINGG ID from the original/baseline model
This allows you to see side-by-side how each record's cluster assignment changed between the two models.
## Enhanced Diff Output with Outer Join
Coming Soon. We're developing an enhanced diff that will provide even more comprehensive comparison capabilities. This enhanced diff will include:
* **All records from both models**, including those that appear in only one output
* **Enhanced null handling**: Intelligent merging of primary key columns when records appear in only one model
## Use Cases
### 1. Field Type Changes
Compare fuzzy vs. exact matching to find the right balance between precision and recall.
### 2. Deterministic Matching Evaluation
See how adding or changing deterministic matching rules affects your clusters.
### 3. Blocking Strategy Comparison
Compare DEFAULT vs. WIDER blocking strategies to understand the performance/accuracy trade-off.
### 4. Model Validation
Validate that model improvements (nicknames, better classifiers, etc.) actually improve results.
### 5. Regression Testing
Ensure that model updates don't accidentally degrade match quality for specific data patterns.
### 6. Configuration Tuning
Experiment with different labelDataSampleSize, numPartitions, or other parameters and see the impact.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Ensuring Scalability
Understanding how blocking is working before running match or link
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
The [Blocking Model](/latest/zmodels) ensures that Zingg stays performant by learning field heuristics and reducing the number of comparisons through the training data. A poorly learnt blocking model will cause Zingg matching jobs to become slow or not complete entirely. This can happen due to a variety of reasons like:
* A user adds significantly larger training samples compared to the labelling learnt by Zingg. The manually added training samples may have the same type of columns and blocking rules learnt are not generic enough. For example, providing California state only training data when the matching is using the State column and data has multiple states.
* When there is a natural bias in the data with lots of null columns used in matching.
* When sufficient labeling has not been done.
* When there a lot of non differentiating columns.
Matching is computationally expensive, and If we can have an understanding of how blocking is working, we can decide whether we need to add more training data. The **verifyBlocking** phase lets us check if we have collected sufficient labels and have a good blocking model.
### The verifyBlocking phase is run as follows:
`./scripts/zingg.sh --phase verifyBlocking --conf >`
The output contains two directories -
`zinggDir/modelId/blocks/timestamp/counts` `zinggDir/modelId/blocks/timestamp/blockSamples`
We can see the counts per block and the top 10% records associated with all the blocks.
For **Zingg Enterprise for Snowflake**, verifyBlocking generates tables with the names:
`zingg_modelId_blocks_timestamp_counts` where we can see the counts per block and `zingg_modelId_blocks_timestamp_blockSamples_hash` where we can see the records associated with the blocks.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Building And Saving The Model
So that the same model can be applied to new data
Builds up the Zingg models using the training data from the above phases and writes them to the folder **zinggDir/modelId** as specified in the config.
```
./zingg.sh --phase train --conf config.json
```
# Finding The Matches
Finds the records that match with each other.
`./zingg.sh --phase match --conf config.json`
As can be seen in the image below, matching records are given the same **z\_cluster** id/Zingg ID in the case of Zingg Enterprise. Each record also gets a **z\_minScore** and **z\_maxScore** which shows the *least/greatest* it matched with other records in the same cluster.
If records across multiple sources have to be matched, the [link phase](/latest/stepbystep/link) should be used.
# Adding Incremental Data
Building a continuously updated identity graph with new, updated, and deleted records
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
Rerunning matching on entire datasets is wasteful, and we lose the lineage of matched records against a persistent identifier. Using the [incremental flow](https://www.learningfromdata.zingg.ai/p/zingg-incremental-flow) feature in [Zingg Enterprise](https://www.zingg.ai), incremental loads can be run to match existing pre-resolved entities. The new and updated records are matched to existing clusters, and new persistent [**ZINGG\_IDs**](https://www.learningfromdata.zingg.ai/p/hello-zingg-id) are generated for records that do not find a match.
If a record gets updated and Zingg Enterprise discovers that it is a more suitable match with another cluster, it will be reassigned. Cluster assignment, merge, and unmerge happens automatically in the flow. Zingg Enterprise also takes care of human feedback on previously matched data to ensure that it does not override the approved records.
### The incremental phase is run as follows:
`./scripts/zingg.sh --phase runIncremental --conf `
### Example incrementalConf.json:
The outputTmp section in the incremental configuration specifies a temporary output location for the results generated during the incremental matching process. This is where Zingg writes the intermediate or temporary results before final processing or merging with the main output.
```json
{
"config" : "config.json",
"incrementalData": [{
"name":"customers_incr",
"format":"csv",
"props": {
"path": "test-incr.csv",
"delimiter": ",",
"header":false
},
"schema": "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, state string, areacode string, dob string, ssn string"
}
],
"outputTmp" :{
"name":"customers_incr_temp",
"format":"csv",
"props": {
"location": "/tmp/zinggOutput_febrl_tmp",
"delimiter": ",",
"header":true
}
}
}
```
### runIncremental can also be triggered using Python by invoking:
`./scripts/zingg.sh --run examples/FebrlExample.py`
#### Python Code Example:
```{python}
#import the packages
from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.ApproverArguments import *
from zinggEC.enterprise.common.IncrementalArguments import *
from zinggEC.enterprise.common.epipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import *
import os
#build the arguments for zingg
args = EArguments()
#set field definitions
recId = EFieldDefinition("recId", "string", MatchType.DONT_USE)
recId.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1","string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
fieldDefs = [recId, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]
args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("/tmp/models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)
#reading dataset into inputPipe and settint it up in 'args'
schema = "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string"
inputPipe = ECsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = ECsvPipe("resultFebrl", "/tmp/febrlOutput")
outputPipe.setHeader("true")
args.setOutput(outputPipe)
#Run findAndLabel
options = ClientOptions([ClientOptions.PHASE,"findAndLabel"])
zingg = EZingg(args, options)
zingg.initAndExecute()
#Run trainMatch after above completes
options = ClientOptions([ClientOptions.PHASE,"trainMatch"])
zingg = EZingg(args, options)
zingg.initAndExecute()
#Now run incremental on output generated above
incrArgs = IncrementalArguments()
incrArgs.setParentArgs(args)
incrPipe = ECsvPipe("testFebrlIncr", "examples/febrl/test-incr.csv", schema)
incrArgs.setIncrementalData(incrPipe)
outputTmpPipe = ECsvPipe("outputTmp", "/tmp/zinggOutput_febrl_tmp")
outputTmpPipe.setHeader("true")
incrArgs.setOutputTmp(outputTmpPipe)
options = ClientOptions([ClientOptions.PHASE,"runIncremental"])
zingg = EZingg(incrArgs, options)
zingg.initAndExecute()
```
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Linking Across Datasets
In many cases like reference data mastering, enrichment, etc, two individual datasets are duplicate-free but they need to be matched against each other. The link phase is used for such scenarios.
`./zingg.sh --phase link --conf config.json`
Sample configuration file [configLink.json](https://github.com/zinggAI/zingg/tree/0.7.0/examples/febrl/configLink.json) is defined at [examples/febrl](https://github.com/zinggAI/zingg/tree/main/examples/febrl). In this option, each record from the first source is matched with all the records from the remaining sources.
The sample output is given in the image below. The linked records are given the same **z\_cluster** id. The last column (**z\_source**) in the output tells the source dataset of that record.
# Lookup Data
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
Sometimes we want to lookup certain records in match output, The lookup feature helps to achieve so. For given lookup records, it assigns zingg-id which decribes which entity cluster it belongs to.
### The lookup phase is run as follows:
`./scripts/zingg.sh --phase runLookup --conf `
### Example lookupConf.json:
```json
{
"config" : "config.json",
"lookupData": [{
"name":"lookup-test-data",
"format":"inMemory"
}
],
"lookupOutput": [
{
"name":"lookup-output",
"format":"csv",
"props": {
"path": "/tmp/zinggOutput/lookup",
"delimiter": ",",
"header":true
}
}
]
}
```
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Approval of Clusters
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
### Coming Soon!
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Reassign ZINGG ID
Reassign the ZINGG IDs for clusters from the original production model
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
## Keep Your IDs Stable Across Changes
When you evolve your Zingg setup—whether upgrading your model, migrating infrastructure, changing data schemas, or moving to new platforms—you want to preserve the ZINGG IDs already flowing through your production systems. The **reassignZinggId** phase intelligently maximizes the preservation of original ZINGG IDs by mapping clusters from your new model back to your original cluster assignments, ensuring continuity and minimizing downstream disruption.
## Why Use Reassign?
* **Model Upgrades**: Add nickname support, improve blocking strategies, or tune classifiers without losing IDs
* **Infrastructure Migration**: Move from one platform to another (e.g., Spark to Snowflake) while maintaining ID continuity
* **Schema Changes**: Update your data schema or field definitions while preserving established IDs
* **Data Migration**: Migrate your data to new systems without disrupting operational workflows
* **Operational Continuity**: Downstream systems continue to work with stable, consistent IDs
* **Intelligent Mapping**: Automatically maximizes the reuse of original ZINGG IDs based on primary key overlap
* **Easy Integration**: No need to update downstream systems when you improve your model
## How It Works
The reassign phase compares two configurations:
1. **Original Configuration**: Your production model with established ZINGG IDs
2. **New Configuration**: Your updated model with improved features, new infrastructure, or schema changes
The reassign phase **maximizes the assignment of original ZINGG IDs** by:
* Identifying clusters in the new output that overlap with original clusters (via primary key matching)
* Reassigning the original ZINGG IDs to matching clusters wherever possible
* Generating new ZINGG IDs only when no match is found in the original clusters
## Usage
### Command Line
```bash
./scripts/zingg.sh --phase reassignZinggId \
--conf examples/febrl/sparkIncremental/configReassign5M.json \
--originalZinggId examples/febrl5M/config.json \
--properties-file config/zingg.conf
```
**Key Parameters:**
* `--conf`: Points to a **wrapper configuration** that references your new/updated model config and specifies the output location for reassigned results
* `--originalZinggId`: Points to your **original production configuration**
* `--properties-file`: Zingg properties file (optional)
**Note:** The `--conf` parameter requires a wrapper configuration (see [Configuration Wrapper](#configuration-wrapper) section below) that includes a `transformedOutputPath` to specify where the reassigned output should be written. This is different from a plain model configuration file.
### Python API
Python users can pass configuration objects directly instead of JSON files.
```python
from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import *
from zinggEC.enterprise.common.TransformedOutputArguments import *
from zinggEC.enterprise.common.EClientOptions import *
# Step 1: Define your ORIGINAL production configuration
# This is your current production setup with established ZINGG IDs
originalArgs = EArguments()
id = EFieldDefinition("id", "string", MatchType.DONT_USE)
id.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
originalArgs.setFieldDefinition([id, fname, lname, stNo, add1, city, areacode, state, dob, ssn])
originalArgs.setModelId("107")
originalArgs.setZinggDir("./models")
originalArgs.setNumPartitions(4)
schema = "id string, fname string, lname string, stNo string, add1 string, city string, areacode string, state string, dob string, ssn string"
originalInputPipe = ECsvPipe("originalData", "examples/febrl5M/febrl_data_5M.csv.gz", schema)
originalArgs.setData(originalInputPipe)
originalOutputPipe = ECsvPipe("originalOutput", "/tmp/zinggOutputOriginal")
originalOutputPipe.setHeader("true")
originalArgs.setOutput(originalOutputPipe)
# IMPORTANT: trainMatch phase must already be completed for the original configuration
# The reassign phase reads the output from the original model
# Step 2: Define your NEW/UPDATED configuration
# Example: Changed some fields from FUZZY to EXACT matching (model upgrade scenario)
newArgs = EArguments()
id = EFieldDefinition("id", "string", MatchType.DONT_USE)
id.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.EXACT) # Changed from FUZZY
add1 = EFieldDefinition("add1", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.EXACT) # Changed from FUZZY
areacode = EFieldDefinition("areacode", "string", MatchType.EXACT) # Changed from FUZZY
state = EFieldDefinition("state", "string", MatchType.EXACT) # Changed from FUZZY
dob = EFieldDefinition("dob", "string", MatchType.EXACT) # Changed from FUZZY
newArgs.setFieldDefinition([id, fname, stNo, add1, city, areacode, state, dob])
newArgs.setModelId("999")
newArgs.setZinggDir("./models")
newArgs.setNumPartitions(4)
newArgs.setLabelDataSampleSize(0.5)
newInputPipe = ECsvPipe("newData", "examples/febrl5M/febrl_data_5M.csv.gz", schema)
newArgs.setData(newInputPipe)
newOutputPipe = ECsvPipe("newOutput", "/tmp/zinggOutputNew")
newOutputPipe.setHeader("true")
newArgs.setOutput(newOutputPipe)
# IMPORTANT: Run trainMatch on the new configuration first
# This generates the output that will be used for reassignment
options = EClientOptions([EClientOptions.PHASE, "trainMatch"])
zinggNew = EZingg(newArgs, options)
zinggNew.initAndExecute()
# Step 3: Create TransformedOutputArguments for reassignZinggId
reassignArgs = TransformedOutputArguments()
reassignArgs.setParentArgs(newArgs) # New configuration
reassignArgs.setOriginalArgs(originalArgs) # Original configuration
# Set output path for reassigned results
reassignOutputPipe = ECsvPipe("reassignedOutput", "/tmp/zinggReassigned")
reassignOutputPipe.setHeader("true")
reassignArgs.setTransformedOutputPath(reassignOutputPipe)
# Step 4: Execute reassignZinggId
# The phase will read outputs from both configurations and maximize original ID preservation
reassignOptions = EClientOptions([EClientOptions.PHASE, "reassignZinggId"])
zinggReassign = EZingg(reassignArgs, reassignOptions)
zinggReassign.initAndExecute()
```
## Configuration Files
### Configuration Wrapper (`examples/febrl/sparkIncremental/configReassign5M.json`)
This wrapper configuration points to your new configuration and specifies where to write the reassigned output:
```json
{
"config": "$ZINGG_ENT_REPO$/spark/examples/febrl5M/configUpdated.json",
"transformedOutputPath": {
"name": "reassignedOutput",
"format": "csv",
"props": {
"location": "/tmp/zinggTransformedOutputReassigned5M",
"delimiter": ",",
"header": true
}
}
}
```
**Note**:
* This wrapper only points to the new configuration. The original configuration is specified via the `--originalZinggId` command-line parameter.
* The `name` field in `transformedOutputPath` can be any arbitrary identifier for the output pipe - it's used internally by Zingg to identify this output destination.
### New Configuration (`examples/febrl5M/configUpdated.json`)
Your updated model configuration with changes—for example, switching some fields from `fuzzy` to `exact` matching:
```json
{
"fieldDefinition": [
{
"fieldName": "id",
"matchType": "dont_use",
"fields": "id",
"dataType": "string",
"primaryKey": true
},
{
"fieldName": "fname",
"matchType": "fuzzy",
"fields": "fname",
"dataType": "string"
},
{
"fieldName": "stNo",
"matchType": "exact",
"fields": "stNo",
"dataType": "string"
},
{
"fieldName": "add1",
"matchType": "fuzzy",
"fields": "add1",
"dataType": "string"
},
{
"fieldName": "city",
"matchType": "exact",
"fields": "city",
"dataType": "string"
},
{
"fieldName": "areacode",
"matchType": "exact",
"fields": "areacode",
"dataType": "string"
},
{
"fieldName": "state",
"matchType": "exact",
"fields": "state",
"dataType": "string"
},
{
"fieldName": "dob",
"matchType": "exact",
"fields": "dob",
"dataType": "string"
}
],
"output": [{
"name": "OUTPUT_FEBRL",
"format": "csv",
"props": {
"location": "/tmp/zinggOutputNew/",
"delimiter": ",",
"header": true
}
}],
"data": [{
"name": "test",
"format": "csv",
"props": {
"location": "examples/febrl5M/febrl_data_5M.csv.gz",
"delimiter": ",",
"header": false
},
"schema": "id string, fname string, lname string, stNo string, add1 string, city string, areacode string, state string, dob string, ssn string"
}],
"outputStats": {
"name": "stats",
"format": "csv",
"props": {
"location": "/tmp/zinggStats_$ZINGG_DYNAMIC_STAT_NAME",
"delimiter": ",",
"header": true
}
},
"labelDataSampleSize": 0.5,
"numPartitions": 4,
"modelId": 999,
"zinggDir": "models"
}
```
### Original Configuration (`examples/febrl5M/config.json`)
Your production configuration with established ZINGG IDs:
```json
{
"fieldDefinition": [
{
"fieldName": "id",
"matchType": "dont_use",
"fields": "id",
"dataType": "string",
"primaryKey": true
},
{
"fieldName": "fname",
"matchType": "fuzzy",
"fields": "fname",
"dataType": "string"
},
{
"fieldName": "lname",
"matchType": "fuzzy",
"fields": "lname",
"dataType": "string"
},
{
"fieldName": "stNo",
"matchType": "fuzzy",
"fields": "stNo",
"dataType": "string"
},
{
"fieldName": "add1",
"matchType": "fuzzy",
"fields": "add1",
"dataType": "string"
},
{
"fieldName": "city",
"matchType": "fuzzy",
"fields": "city",
"dataType": "string"
},
{
"fieldName": "areacode",
"matchType": "fuzzy",
"fields": "areacode",
"dataType": "string"
},
{
"fieldName": "state",
"matchType": "fuzzy",
"fields": "state",
"dataType": "string"
},
{
"fieldName": "dob",
"matchType": "fuzzy",
"fields": "dob",
"dataType": "string"
},
{
"fieldName": "ssn",
"matchType": "fuzzy",
"fields": "ssn",
"dataType": "string"
}
],
"output": [{
"name": "output",
"format": "csv",
"props": {
"location": "/tmp/zinggOutputOriginal",
"delimiter": ",",
"header": true
}
}],
"data": [{
"name": "test",
"format": "csv",
"props": {
"location": "examples/febrl5M/febrl_data_5M.csv.gz",
"delimiter": ",",
"header": false
},
"schema": "id string, fname string, lname string, stNo string, add1 string, city string, areacode string, state string, dob string, ssn string"
}],
"outputStats": {
"name": "stats",
"format": "csv",
"props": {
"location": "/tmp/zinggStats_$ZINGG_DYNAMIC_STAT_NAME",
"delimiter": ",",
"header": true
}
},
"labelDataSampleSize": 0.01,
"numPartitions": 4000,
"modelId": 107,
"zinggDir": "models"
}
```
## Output Format
The output is written to the `transformedOutputPath` specified in your configuration and contains the same structure as the new model's output, but with reassigned ZINGG IDs that maximize preservation of the original IDs.
## Use Cases
### 1. Model Upgrades
When you add new features like nicknames, better blocking strategies, or improved classifiers, use reassignZinggId to maintain ID continuity.
### 2. Infrastructure Migration
Moving from Spark to Snowflake, or vice versa? Reassign helps you maintain the same ZINGG IDs across platforms.
### 3. Schema Evolution
If your data schema changes (new fields, removed fields, different field types), reassignZinggId preserves IDs for records that can still be matched via primary key.
### 4. Data Migration
Migrating from Databricks to Fabric, or vice versa? When migrating data to new systems or consolidating data sources, maintain ID consistency across the migration.
### 5. Platform Upgrades
Upgrading Spark versions, Snowflake features, or other platform components? Keep your IDs stable through the upgrade process.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Data Sources and Sinks
Zingg *connects, reads,* and *writes* to most on-premise and cloud data sources.
Zingg can read and write to **Databricks, Microsoft Fabric, Snowflake, Cassandra, S3, Azure, Elastic, Exasol**, major **RDBMS**, and any **Spark**-supported data sources.\
\
Zingg also works with all major file formats like Parquet, Avro, JSON, XLSX, CSV, TSV, etc. This is done through the Zingg [Pipe](/latest/connectors/pipes) abstraction.
# Zingg Pipes
Zingg Pipes are an *abstraction* for a data source from which Zingg fetches data for matching or to which Zingg writes its output. This lets users connect to literally any datastore that has a Spark connector.
The pipe is an easy way to specify *properties* and *formats* for the Spark connector of the relevant data source. Zingg pipes can be configured through the config [JSON](/latest/stepbystep/configuration) passed to the program by outlining the datastore connection properties.
Pipes can be configured for the data or the output attributes on the [JSON](/latest/stepbystep/configuration).
Each pipe has the following attributes:
## name
A unique name to identify the data store.
## format
One of the Spark-supported connector formats - jdbc/avro/parquet etc.
## options
Properties to be passed to **spark.read** and **spark.write.**
Let us look at some common data sources and their configurations.
# Databricks
As a Spark-based application, the **Zingg Community version** works seamlessly on Databricks. Zingg leverages Databricks' Spark environment.
You can access all the supported data sources like Parquet, Delta tables, and Unity Catalog (for governed Delta tables) while running **Zingg Community version**.
**Zingg Enterprise version** provides additional advanced features and support.
Please check the various ways in which you can run **Zingg Community version on Databricks** [here](/latest/running/databricks)
# Microsoft Fabric
As a Spark-based application, the **Zingg Community version** works seamlessly on Microsoft Fabric. Zingg leverages Fabric's Spark environment.
You can access all the supported data sources like parquet and other file formats while running **Zingg Community version**. **Zingg Enterprise version** supports additional advanced features.
Please check the various ways in which you can run **Zingg Community version on Microsoft Fabric** [here](/latest/running/running-on-fabric)
# Snowflake
Identity Resolution on Snowflake
Check a step-by-step tutorial at [Identity Resolution on Snowflake with open source Zingg](https://www.zingg.ai/documentation-article/identifying-duplicates-in-snowflake-with-zingg).
The config value for the data and output attributes of the JSON is:
```json
"data" : [ {
"name":"test",
"format":"net.snowflake.spark.snowflake",
"props": {
"sfUrl": "rfa59271.snowflakecomputing.com",
"sfUser": "sonalgoyal",
"sfPassword":"ZZ",
"sfDatabase":"TEST",
"sfSchema":"PUBLIC",
"sfWarehouse":"COMPUTE_WH",
"dbtable": "FEBRL",
"application":"zingg_zingg"
}
} ]
```
One must include Snowflake JDBC driver and Spark dependency on the classpath. The jars can be downloaded from the **maven** repository ([1](https://mvnrepository.com/artifact/net.snowflake/snowflake-jdbc), [2](https://mvnrepository.com/artifact/net.snowflake/spark-snowflake)).
```
spark.jars=snowflake-jdbc-3.13.19.jar,spark-snowflake_2.12-2.10.0-spark_3.1.jar
```
For Zingg to discover the Snowflake jars, please add the property **spark.jars** in [Zingg's runtime properties.](/latest/stepbystep/zingg-runtime-properties)
If you are looking for a native-run on Snowflake without using Spark, check [Zingg Enterprise Snowflake](https://www.zingg.ai/company/zingg-enterprise-snowflake).
# JDBC
Zingg can connect to various databases such as MySQL, DB2, MariaDB, MS SQL, Oracle, PostgreSQL, etc. using JDBC. One just needs to download the appropriate driver and made it accessible to the application.
To include the JDBC driver for your particular database on the Spark classpath, please add the property **spark.jars** in [Zingg's runtime properties.](/latest/stepbystep/zingg-runtime-properties)
```
spark.jars=
```
Connection details are given in the following sections for a few common JDBC sources.
# Postgres
## JSON Settings For Reading Data From Postgres Database:
```json
"data" : [{
"name":"test",
"format":"jdbc",
"props": {
"url": "jdbc:postgresql://localhost:5432/",
"dbtable": "",
"driver": "org.postgresql.Driver",
"user": "",
"password": ""
}
}],
```
Replace `` with the *name* of the database in addition to other props. For more details, refer to the [Spark documentation](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).
# MySQL
## Reading From MySQL Database:
```json
"data" : [{
"name":"test",
"format":"jdbc",
"props": {
"url": "jdbc:mysql://localhost:3306/",
"dbtable": "testData",
"driver": "com.mysql.cj.jdbc.Driver",
"user": "root",
"password": "password"
}
}],
```
Please replace `` with the *name* of the database in addition to other props. For more details, refer to the [Spark documentation](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).
# AWS S3
Zingg can use AWS S3 as a source and sink
## Steps to run zingg on S3
* Set a bucket, for example - *zingg28032023* and a folder inside it,for example - *zingg*
* Create aws access key and export via env vars (ensure that the user with below keys has read/write access to above)\
`export AWS_ACCESS_KEY_ID=`\
`export AWS_SECRET_ACCESS_KEY=`\
(if mfa is enabled AWS\_SESSION\_TOKEN env var would also be needed )
* Download *hadoop-aws-3.1.0.jar* and *aws-java-sdk-bundle-1.11.271.jar* via maven
* Set above in zingg.conf\
spark.jars=//hadoop-aws-3.1.0.jar,//aws-java-sdk-bundle-1.11.271.jar
* Run using below commands
```bash
./scripts/zingg.sh --phase findTrainingData --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase label --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase train --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase match --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
```
## Model location
```
Models etc. would get saved in
Amazon S3 > Buckets > zingg28032023 > zingg > 100
```
# Cassandra
```json
"output" : [
{
"name":"sampleTest",
"format":"CASSANDRA" ,
"props": {
"table":"dataschematest",
"keyspace":"zingg",
"cluster":"zingg",
"spark.cassandra.connection.host":"192.168.0.6"
},
"sparkProps": {
"spark.cassandra.connection.host":"127.0.0.1"
},
"mode":"Append"
}
]
```
# MongoDB
```json
"data" : [{
"name":"mongodb",
"format":"mongo",
"props": {
"uri": "mongodb://127.0.0.1/people.contacts"
}
}]
```
# Neo4j
```json
"data" : [{
"name":"neo",
"format":"org.neo4j.spark.DataSource",
"props": {
"url": "bolt://localhost:7687",
"labels":"Person"
}
}]
```
# Parquet
```json
"data" : [{
"name":"parquetFiles",
"format":"parquet",
"props": {
"path": "/home/zingg"
}
}]
```
# BigQuery
Zingg can seamlessly work with Google BigQuery. Please find below details about the properties that must be set.
The two driver jars namely **spark-bigquery-with-dependencies\_2.12-0.24.2.jar** and **gcs-connector-hadoop2-latest.jar** are required to work with BigQuery. To include these BigQuery drivers in the classpath, please configure the [runtime properties](/latest/stepbystep/zingg-runtime-properties) to include these
```bash
spark.jars=./spark-bigquery-with-dependencies_2.12-0.24.2.jar,./gcs-connector-hadoop2-latest.jar
```
In addition, the following property needs to be set
```bash
spark.hadoop.fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem
```
If Zingg is run from outside the Google cloud, it requires authentication, please set the following *environment variable* to the location of the file containing the *service account key*. A service account key can be created and downloaded in JSON format from the [Google Cloud console](https://cloud.google.com/docs/authentication/getting-started).
```bash
export GOOGLE_APPLICATION_CREDENTIALS=path to google service account key file
```
Connection properties for BigQuery as a data source and data sink are given below. If you are curious to know more about how Spark connects to BigQuery, you may look at the [Spark BigQuery connector documentation](https://github.com/GoogleCloudDataproc/spark-bigquery-connector).
## Properties for reading data from BigQuery:
The property `credentialsFile` should point to the Google service account key file location. This is the same path that is used to set variable `GOOGLE_APPLICATION_CREDENTIALS`. The `table` property should point to a BigQuery table that contains source data. The property `viewsEnabled` must be set to **true** only.
```json
"data" : [{
"name":"test",
"format":"bigquery",
"props": {
"credentialsFile": "/home/work/product/final/zingg-1/mynotification-46566-905cbfd2723f.json",
"table": "mynotification-46566.zinggdataset.zinggtest",
"viewsEnabled": true
}
}],
```
## Properties For Writing Data To BigQuery:
To write to BigQuery, a bucket needs to be created and assigned to the `temporaryGcsBucket` property.
```json
"output" : [{
"name":"output",
"format":"bigquery",
"props": {
"credentialsFile": "/home/work/product/final/zingg-1/mynotification-46566-905cbfd2723f.json",
"table": "mynotification-46566.zinggdataset.zinggOutput",
"temporaryGcsBucket":"zingg-test",
}
}],
```
## Notes:
* The library **"gcs-connector-hadoop2-latest.jar"** can be downloaded from [Google](https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop2-latest.jar) and the library **"spark-bigquery-with-dependencies\_2.12-0.24.2"** from the [maven repo](https://repo1.maven.org/maven2/com/google/cloud/spark/spark-bigquery-with-dependencies_2.12/0.24.2/spark-bigquery-with-dependencies_2.12-0.24.2.jar).
* A typical service account key file looks like below (JSON).
```json
{
"type": "service_account",
"project_id": "mynotification-46566",
"private_key_id": "905cbfd273ff9205d1ce06fa6908e54534",
"private_key": "-----BEGIN PRIVATE KEY-----CERT.....",
"client_id": "11143646541215487",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/zingtest%44mynotification-46566.iam.gserviceaccount.com"
}
```
# Exasol
[Exasol](https://www.exasol.com/) is a in-memory database built for analytics. You can use it to interact with Zingg AI.
## Prerequisites
To use the Exasol database with Zingg, first you should download the required dependencies and set them in the Zingg configuration file.
### Download Assembled Exasol Spark Connector Jar
Please download the latest assembled Exasol [spark-connector](https://github.com/exasol/spark-connector/releases) jar file. Go to the `Assets` section below, and make sure you download the jar file with `-assembly` suffix in the file name.
### Update Zingg Configuration
After downloading the Exasol spark-connector jar file, you should update the `spark.jars` parameter in [Zingg's runtime properties.](/latest/stepbystep/zingg-runtime-properties) so that it can find the Exasol dependencies.
For example:
```
spark.jars=spark-connector_2.12-1.3.0-spark-3.3.2-assembly.jar
```
If there are more than one jar files, please use *comma* as separator. Additionally, please change the version accordingly so that it matches your Zingg and Spark versions.
## Connector Settings
Finally, create a configuration JSON file for Zingg, and update the `data` or `output` settings accordingly.
For example:
```json
...
"data": [
{
"name": "input",
"format": "com.exasol.spark",
"props": {
"host": "10.11.0.2",
"port": "8563",
"username": "sys",
"password": "exasol",
"query": "SELECT * FROM DB_SCHEMA.CUSTOMERS"
}
}
],
...
```
Similarly, for output:
```json
...
"output": [
{
"name": "output",
"format": "com.exasol.spark",
"props": {
"host": "10.11.0.2",
"port": "8563",
"username": "sys",
"password": "exasol",
"create_table": "true",
"table": "DB_SCHEMA.ENTITY_RESOLUTION",
},
"mode": "Append"
}
],
...
```
Please note that, the `host` parameter should be the first internal node's **IPv4** **address**.
As Zingg uses [Exasol Spark connector](https://github.com/exasol/spark-connector) underneath, please also check out the [user guide](https://github.com/exasol/spark-connector/blob/main/doc/user_guide/user_guide.md) and [configuration options](https://github.com/exasol/spark-connector/blob/main/doc/user_guide/user_guide.md#configuration-options) for more information.
# Working With Python
A whole new way to work with Zingg!
Instead of configuring Zingg using JSON, we can now use Python to build and run Zingg entity and identity resolution programs. This is handy when you want to run Zingg on an existing Spark cluster. To run on a local machine, please install from the release before running Zingg Python programs.
The Zingg Python package can be installed by invoking:
`python -m pip install zingg`
Detailed documentation of the Python API is available at:
Example programs for Python exist under [examples](https://github.com/zinggAI/zingg/tree/main/examples/febrl). Please check [examples/febrl/FebrlExample.py](https://github.com/zinggAI/zingg/blob/main/examples/febrl/FebrlExample.py) to get started.
Please refer to the [command line guide](/latest/stepbystep/zingg-command-line) for running Python programs. Please note that Zingg Python programs are PySpark programs and hence need the Zingg CLI to execute.
# Community Python API
## Community Zingg Entity Resolution Python Package
Community Zingg Python APIs for entity resolution, identity resolution, record linkage, data mastering and deduplication using ML ()
**NOTE**
Requires python 3.6+; spark 3.5.0 Otherwise, [`zingg.client.Zingg()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.client.Zingg.md#zingg.client.Zingg) cannot be executed
## API Documentation
* [zingg.client module](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/zingg.client.md)
* [Classes](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/zingg.client.md#classes)
* [zingg.client.Zingg](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.client.Zingg.md)
* [zingg.client.ZinggWithSpark](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.client.ZinggWithSpark.md)
* [zingg.client.Arguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.client.Arguments.md)
* [zingg.client.ClientOptions](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.client.ClientOptions.md)
* [zingg.client.FieldDefinition](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.client.FieldDefinition.md)
* [zingg.pipes module](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/zingg.pipes.md)
* [Classes](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/zingg.pipes.md#classes)
* [zingg.pipes.Pipe](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.pipes.Pipe.md)
* [zingg.pipes.CsvPipe](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.pipes.CsvPipe.md)
* [zingg.pipes.BigQueryPipe](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.pipes.BigQueryPipe.md)
* [zingg.pipes.SnowflakePipe](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonOss/markdown/_autosummary/zingg.pipes.SnowflakePipe.md)
## Example API Usage
```python
from zingg.client import *
from zingg.pipes import *
#build the arguments for zingg
args = Arguments()
#set field definitions
fname = FieldDefinition("fname", "string", MatchType.FUZZY)
lname = FieldDefinition("lname", "string", MatchType.FUZZY)
stNo = FieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = FieldDefinition("add1","string", MatchType.FUZZY)
add2 = FieldDefinition("add2", "string", MatchType.FUZZY)
city = FieldDefinition("city", "string", MatchType.FUZZY)
areacode = FieldDefinition("areacode", "string", MatchType.FUZZY)
state = FieldDefinition("state", "string", MatchType.FUZZY)
dob = FieldDefinition("dob", "string", MatchType.FUZZY)
ssn = FieldDefinition("ssn", "string", MatchType.FUZZY)
fieldDefs = [fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]
args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)
#reading dataset into inputPipe and setting it up in 'args'
schema = "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string"
inputPipe = CsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = CsvPipe("resultFebrl", "/tmp/febrlOutput")
args.setOutput(outputPipe)
options = ClientOptions([ClientOptions.PHASE,"match"])
#Zingg execution for the given phase
zingg = Zingg(args, options)
zingg.initAndExecute()
```
# Enterprise Python API
A whole new way to work with Zingg Enterprise!
## Example API Usage
```python
from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.ApproverArguments import *
from zinggEC.enterprise.common.IncrementalArguments import *
from zinggEC.enterprise.common.MappingMatchType import *
from zinggEC.enterprise.common.epipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import *
import os
#build the arguments for zingg
args = EArguments()
#set field definitions
recId = EFieldDefinition("recId", "string", MatchType.DONT_USE)
recId.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
# for mapping match type
#fname = EFieldDefinition("fname", "string", MatchType.FUZZY, MappingMatchType("MAPPING", "NICKNAMES_TEST"))
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1","string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
fieldDefs = [recId, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]
args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("./models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)
# Set the blocking strategy for the Zingg Model as either DEFAULT or WIDER - if you do not set anything, the model follows DEFAULT strategy
args.setBlockingModel("DEFAULT")
#setting pass thru condition
args.setPassthroughExpr("fname = 'matilda'")
#setting deterministic matching conditions
dm1 = DeterministicMatching('fname','stNo','add1')
dm2 = DeterministicMatching('ssn')
dm3 = DeterministicMatching('fname','stNo','lname')
args.setDeterministicMatchingCondition(dm1,dm2,dm3)
#reading dataset into inputPipe and setting it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string"
inputPipe = ECsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = ECsvPipe("resultFebrl", "/tmp/febrlOutput")
outputPipe.setHeader("true")
args.setOutput(outputPipe)
# Zingg execution for the given phase
# options = ClientOptions([ClientOptions.PHASE,"findAndLabel"])
options = ClientOptions([ClientOptions.PHASE,"trainMatch"])
zingg = EZingg(args, options)
zingg.initAndExecute()
incrArgs = IncrementalArguments()
incrArgs.setParentArgs(args)
incrPipe = ECsvPipe("testFebrlIncr", "examples/febrl/test-incr.csv", schema)
incrArgs.setIncrementalData(incrPipe)
incrOptions = ClientOptions([ClientOptions.PHASE,"runIncremental"])
zinggIncr = EZingg(incrArgs, incrOptions)
zinggIncr.initAndExecute()
```
# ZinggEC Python API
## Zingg Enterpise Entity Resolution Python Package
Zingg Enterprise Python APIs for entity resolution, identity resolution, record linkage, data mastering and deduplication using ML ()
**NOTE**
Requires python 3.6+; spark 3.5.0 Otherwise, `zinggES.enterprise.spark.ESparkClient()` cannot be executed
* [Zingg Enterprise Common Package](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/zinggEC.md)
* [ApproverArguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md)
* [zinggEC.enterprise.common.ApproverArguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggec-enterprise-common-approverarguments)
* [`ApproverArguments`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments)
* [`ApproverArguments.getApprovalQuery()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.getApprovalQuery)
* [`ApproverArguments.getArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.getArgs)
* [`ApproverArguments.getDestination()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.getDestination)
* [`ApproverArguments.getParentArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.getParentArgs)
* [`ApproverArguments.setApprovalQuery()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.setApprovalQuery)
* [`ApproverArguments.setArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.setArgs)
* [`ApproverArguments.setDestination()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.setDestination)
* [`ApproverArguments.setParentArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/ApproverArguments.md#zinggEC.enterprise.common.ApproverArguments.ApproverArguments.setParentArgs)
* [IncrementalArguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md)
* [zinggEC.enterprise.common.IncrementalArguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggec-enterprise-common-incrementalarguments)
* [`IncrementalArguments`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments)
* [`IncrementalArguments.getArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.getArgs)
* [`IncrementalArguments.getDeleteAction()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.getDeleteAction)
* [`IncrementalArguments.getDeletedData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.getDeletedData)
* [`IncrementalArguments.getIncrementalData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.getIncrementalData)
* [`IncrementalArguments.getOutputTmp()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.getOutputTmp)
* [`IncrementalArguments.getParentArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.getParentArgs)
* [`IncrementalArguments.setArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.setArgs)
* [`IncrementalArguments.setDeleteAction()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.setDeleteAction)
* [`IncrementalArguments.setDeletedData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.setDeletedData)
* [`IncrementalArguments.setIncrementalData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.setIncrementalData)
* [`IncrementalArguments.setOutputTmp()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.setOutputTmp)
* [`IncrementalArguments.setParentArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/IncrementalArguments.md#zinggEC.enterprise.common.IncrementalArguments.IncrementalArguments.setParentArgs)
* [MappingMatchType](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/MappingMatchType.md)
* [zinggEC.enterprise.common.MappingMatchType](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/MappingMatchType.md#zinggec-enterprise-common-mappingmatchtype)
* [`MappingMatchType`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/MappingMatchType.md#zinggEC.enterprise.common.MappingMatchType.MappingMatchType)
* [`MappingMatchType.getMappingMatchType()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/MappingMatchType.md#zinggEC.enterprise.common.MappingMatchType.MappingMatchType.getMappingMatchType)
* [epipes](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md)
* [zinggEC.enterprise.common.epipes](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggec-enterprise-common-epipes)
* [`ECsvPipe`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.ECsvPipe)
* [`ECsvPipe.setDelimiter()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.ECsvPipe.setDelimiter)
* [`ECsvPipe.setHeader()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.ECsvPipe.setHeader)
* [`ECsvPipe.setLocation()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.ECsvPipe.setLocation)
* [`EPipe`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.EPipe)
* [`EPipe.getPassthroughExpr()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.EPipe.getPassthroughExpr)
* [`EPipe.getPassthruData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.EPipe.getPassthruData)
* [`EPipe.getUsableData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.EPipe.getUsableData)
* [`EPipe.hasPassThru()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.EPipe.hasPassThru)
* [`EPipe.setPassthroughExpr()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.EPipe.setPassthroughExpr)
* [`InMemoryPipe`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.InMemoryPipe)
* [`InMemoryPipe.getDataset()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.InMemoryPipe.getDataset)
* [`InMemoryPipe.setDataset()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.InMemoryPipe.setDataset)
* [`UCPipe`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.UCPipe)
* [`UCPipe.setTable()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/epipes.md#zinggEC.enterprise.common.epipes.UCPipe.setTable)
* [EArguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md)
* [zinggEC.enterprise.common.EArguments](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggec-enterprise-common-earguments)
* [`DeterministicMatching`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.DeterministicMatching)
* [`DeterministicMatching.getDeterministicMatching()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.DeterministicMatching.getDeterministicMatching)
* [`EArguments`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments)
* [`EArguments.getArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getArgs)
* [`EArguments.getData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getData)
* [`EArguments.getDeterministicMatching()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getDeterministicMatching)
* [`EArguments.getFieldDefinition()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getFieldDefinition)
* [`EArguments.getOutputStats()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getOutputStats)
* [`EArguments.getPassthroughExpr()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getPassthroughExpr)
* [`EArguments.getPrimaryKey()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.getPrimaryKey)
* [`EArguments.setArgs()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setArgs)
* [`EArguments.setBlockingModel()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setBlockingModel)
* [`EArguments.setData()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setData)
* [`EArguments.setDeterministicMatchingCondition()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setDeterministicMatchingCondition)
* [`EArguments.setFieldDefinition()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setFieldDefinition)
* [`EArguments.setOutputStats()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setOutputStats)
* [`EArguments.setPassthroughExpr()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EArguments.md#zinggEC.enterprise.common.EArguments.EArguments.setPassthroughExpr)
* [EFieldDefinition](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EFieldDefinition.md)
* [zinggEC.enterprise.common.EFieldDefinition](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EFieldDefinition.md#zinggec-enterprise-common-efielddefinition)
* [`EFieldDefinition`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EFieldDefinition.md#zinggEC.enterprise.common.EFieldDefinition.EFieldDefinition)
* [`EFieldDefinition.getMatchTypeArray()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EFieldDefinition.md#zinggEC.enterprise.common.EFieldDefinition.EFieldDefinition.getMatchTypeArray)
* [`EFieldDefinition.getPrimaryKey()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EFieldDefinition.md#zinggEC.enterprise.common.EFieldDefinition.EFieldDefinition.getPrimaryKey)
* [`EFieldDefinition.setPrimaryKey()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/EFieldDefinition.md#zinggEC.enterprise.common.EFieldDefinition.EFieldDefinition.setPrimaryKey)
## API Reference
* [Module Index](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/py-modindex.md)
* [Index](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/genindex.md)
* [Search Page](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonEC/markdown/search.md)
## Example API Usage
```python
from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.ApproverArguments import *
from zinggEC.enterprise.common.IncrementalArguments import *
from zinggEC.enterprise.common.MappingMatchType import *
from zinggEC.enterprise.common.epipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import *
import os
#build the arguments for zingg
args = EArguments()
#set field definitions
recId = EFieldDefinition("recId", "string", MatchType.DONT_USE)
recId.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
# for mapping match type
#fname = EFieldDefinition("fname", "string", MatchType.FUZZY, MappingMatchType("MAPPING", "NICKNAMES_TEST"))
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1","string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
fieldDefs = [recId, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]
args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("./models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)
# Set the blocking strategy for the Zingg Model as either DEFAULT or WIDER - if you do not set anything, the model follows DEFAULT strategy
args.setBlockingModel("DEFAULT")
#setting pass thru condition
args.setPassthroughExpr("fname = 'matilda'")
#setting deterministic matching conditions
dm1 = DeterministicMatching('fname','stNo','add1')
dm2 = DeterministicMatching('ssn')
dm3 = DeterministicMatching('fname','stNo','lname')
args.setDeterministicMatchingCondition(dm1,dm2,dm3)
#reading dataset into inputPipe and setting it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string"
inputPipe = ECsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = ECsvPipe("resultFebrl", "/tmp/febrlOutput")
outputPipe.setHeader("true")
args.setOutput(outputPipe)
# Zingg execution for the given phase
# options = ClientOptions([ClientOptions.PHASE,"findAndLabel"])
options = ClientOptions([ClientOptions.PHASE,"trainMatch"])
zingg = EZingg(args, options)
zingg.initAndExecute()
incrArgs = IncrementalArguments()
incrArgs.setParentArgs(args)
incrPipe = ECsvPipe("testFebrlIncr", "examples/febrl/test-incr.csv", schema)
incrArgs.setIncrementalData(incrPipe)
incrOptions = ClientOptions([ClientOptions.PHASE,"runIncremental"])
zinggIncr = EZingg(incrArgs, incrOptions)
zinggIncr.initAndExecute()
```
# ZinggES Python API
## Zingg Enterpise Entity Resolution Python Package
Zingg Enterprise Python APIs for entity resolution, identity resolution, record linkage, data mastering and deduplication using ML ()
**NOTE**
Requires python 3.6+; spark 3.5.0 Otherwise, [`zinggES.enterprise.spark.ESparkClient()`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/ESparkClient.md#module-zinggES.enterprise.spark.ESparkClient) cannot be executed
* [Zingg Enterprise Spark Package](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/zinggES.md)
* [ESparkClient](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/ESparkClient.md)
* [zinggES.enterprise.spark.ESparkClient](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/ESparkClient.md#zingges-enterprise-spark-esparkclient)
* [`EZingg`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/ESparkClient.md#zinggES.enterprise.spark.ESparkClient.EZingg)
* [`EZinggWithSpark`](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/ESparkClient.md#zinggES.enterprise.spark.ESparkClient.EZinggWithSpark)
## API Reference
* [Module Index](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/py-modindex.md)
* [Index](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/genindex.md)
* [Search Page](https://github.com/zinggAI/zingg/tree/0.7.0/docs/pythonES/markdown/search.md)
## Example API Usage
```python
from zingg.client import *
from zingg.pipes import *
from zinggEC.enterprise.common.ApproverArguments import *
from zinggEC.enterprise.common.IncrementalArguments import *
from zinggEC.enterprise.common.MappingMatchType import *
from zinggEC.enterprise.common.epipes import *
from zinggEC.enterprise.common.EArguments import *
from zinggEC.enterprise.common.EFieldDefinition import EFieldDefinition
from zinggES.enterprise.spark.ESparkClient import *
import os
#build the arguments for zingg
args = EArguments()
#set field definitions
recId = EFieldDefinition("recId", "string", MatchType.DONT_USE)
recId.setPrimaryKey(True)
fname = EFieldDefinition("fname", "string", MatchType.FUZZY)
# for mapping match type
#fname = EFieldDefinition("fname", "string", MatchType.FUZZY, MappingMatchType("MAPPING", "NICKNAMES_TEST"))
lname = EFieldDefinition("lname", "string", MatchType.FUZZY)
stNo = EFieldDefinition("stNo", "string", MatchType.FUZZY)
add1 = EFieldDefinition("add1","string", MatchType.FUZZY)
add2 = EFieldDefinition("add2", "string", MatchType.FUZZY)
city = EFieldDefinition("city", "string", MatchType.FUZZY)
areacode = EFieldDefinition("areacode", "string", MatchType.FUZZY)
state = EFieldDefinition("state", "string", MatchType.FUZZY)
dob = EFieldDefinition("dob", "string", MatchType.FUZZY)
ssn = EFieldDefinition("ssn", "string", MatchType.FUZZY)
fieldDefs = [recId, fname, lname, stNo, add1, add2, city, areacode, state, dob, ssn]
args.setFieldDefinition(fieldDefs)
#set the modelid and the zingg dir
args.setModelId("100")
args.setZinggDir("./models")
args.setNumPartitions(4)
args.setLabelDataSampleSize(0.5)
# Set the blocking strategy for the Zingg Model as either DEFAULT or WIDER - if you do not set anything, the model follows DEFAULT strategy
args.setBlockingModel("DEFAULT")
#setting pass thru condition
args.setPassthroughExpr("fname = 'matilda'")
#setting deterministic matching conditions
dm1 = DeterministicMatching('fname','stNo','add1')
dm2 = DeterministicMatching('ssn')
dm3 = DeterministicMatching('fname','stNo','lname')
args.setDeterministicMatchingCondition(dm1,dm2,dm3)
#reading dataset into inputPipe and setting it up in 'args'
#below line should not be required if you are reading from in memory dataset
#in that case, replace df with input df
schema = "recId string, fname string, lname string, stNo string, add1 string, add2 string, city string, areacode string, state string, dob string, ssn string"
inputPipe = ECsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
outputPipe = ECsvPipe("resultFebrl", "/tmp/febrlOutput")
outputPipe.setHeader("true")
args.setOutput(outputPipe)
# Zingg execution for the given phase
# options = ClientOptions([ClientOptions.PHASE,"findAndLabel"])
options = ClientOptions([ClientOptions.PHASE,"trainMatch"])
zingg = EZingg(args, options)
zingg.initAndExecute()
incrArgs = IncrementalArguments()
incrArgs.setParentArgs(args)
incrPipe = ECsvPipe("testFebrlIncr", "examples/febrl/test-incr.csv", schema)
incrArgs.setIncrementalData(incrPipe)
incrOptions = ClientOptions([ClientOptions.PHASE,"runIncremental"])
zinggIncr = EZingg(incrArgs, incrOptions)
zinggIncr.initAndExecute()
```
# Running Zingg On Cloud
# Running On AWS
One option is to use the `spark-submit` option with the Zingg config and phase. Please note that the **config.json** should be available locally at the driver for Zingg to use it.
`````
aws emr create-cluster --name "Add Spark Step Cluster" --release-label emr-6.2.0 --applications Name=Zingg \
--ec2-attributes KeyName=myKey --instance-type --instance-count \
--steps Type=Spark,Name="Zingg",ActionOnFailure=CONTINUE,Args=[--class,zingg.spark.client.SparkClient,,--phase,,--conf,] --use-default-roles````
`````
A step-by-step is provided [here](https://blog.infostrux.com/identity-resolution-with-zingg-ai-snowflake-and-aws-emr-for-the-canadian-football-league-22cf0850ab53). The guide mentions training locally using Zingg Docker, but the **findTrainingData** and **label** phases can be executed on EMR directly.
A second option is to run Zingg Python code in [AWS EMR Notebooks](https://aws.amazon.com/emr/features/notebooks/)
# Running On Azure
To run Zingg on Azure, please refer to [Databricks configuration](/latest/running/databricks).
# Running On Databricks
Step by Step Identity Resolution with Zingg on Databricks
You can run the **Zingg Community Version** on Databricks directly using the Databricks notebook interface.
## For running the Zingg Community version on Databricks:
This uses the Zingg Community Python API and supports Delta tables as well as Unity Catalog (for governed Delta tables) using UCPipe. An [example notebook](https://github.com/zinggAI/zingg/tree/0.7.0/examples/databricks/FebrlExample.ipynb) and a [step by step guide](https://www.zingg.ai/documentation-article/identity-resolution-on-databricks-for-customer-360) is available.
## For running the Zingg Enterprise version on Databricks:
This uses the Zingg Enterprise Python API and all [file formats and data sources and sinks](https://github.com/zinggAI/zingg/tree/0.7.0/docs/dataSourcesAndSinks/README.md) are supported within Databricks with **Zingg Enterprise** including Delta tables, Unity Catalog (for governed Delta tables), and InMemory.
# Running on Fabric
Step by Step Identity Resolution with Zingg on Fabric
You can run the **Zingg Community Version** on Microsoft Fabric directly using the Fabric notebook interface.
## For running the Zingg Community version on Microsoft Fabric:
This uses the Zingg Community Python API and an [example notebook](https://github.com/zinggAI/zingg/tree/0.7.0/examples/fabric/ExampleNotebook.ipynb) and a [step by step guide](https://www.zingg.ai/documentation-article/step-by-step-identity-resolution-with-zingg-on-fabric) is available.
## For running the Zingg Enterprise version on Microsoft Fabric:
This uses the Zingg Enterprise Python API and all [file formats and data sources and sinks](https://github.com/zinggAI/zingg/tree/0.7.0/docs/dataSourcesAndSinks/README.md) are supported within Microsoft Fabric with **Zingg Enterprise** including additional advanced features and support.
# Zingg Models
Zingg learns two models from the data.
## 1. Blocking Model
One fundamental problem with scaling data mastering is that the number of comparisons increases **quadratically** as the number of input records increases.
Zingg learns a clustering/blocking model which indexes near similar records. This means that Zingg does not compare every record with every other record. Typical Zingg comparisons are **0.05-1%** of the possible problem space.
## 2. Similarity Model
The similarity model helps Zingg to predict which record pairs match. The similarity is run only on records within the same block/cluster to scale the problem to larger datasets. The similarity model is a classifier that predicts the similarity of records that are not exactly the same but could belong together.
To build these models, training data is needed. Zingg comes with an interactive learner to rapidly build training sets.
# Pre-Trained Models
Zingg comes with pre-trained models for the Febrl, North Carolina Voters, Amazon Google products and few other datasets under the [models](https://github.com/zinggAI/zingg/tree/main/models) folder.
Information on which model is for which dataset is present in [**modelIds.txt**](https://github.com/zinggAI/zingg/blob/main/examples/modelIds.txt) under the models folder.
# Improving Accuracy
# Ignoring Commonly Occuring Words While Matching
Common words like Mr, Pvt, Av, St, Street etc. do not add differential signals and confuse matching. These words are called **stopwords** and matching is more accurate when stopwords are ignored.
The stopwords can be recommended by Zingg by invoking:
`./scripts/zingg.sh --phase recommend --conf --column `
The stopwords generated are stored at the location - models/100/stopWords/columnName. This will give you the list of stopwords along with their frequency.
By default, Zingg extracts 10% of the high-frequency unique words from a dataset. If the user wants a different selection, they should set up the following property in the config file under the respective field for which they want stopwords:
```
stopWordsCutoff:
```
Once you have verified the above stop words, you can configure them in the JSON variable **stopWords** with the path to the CSV file containing them. Please ensure while editing the CSV or building it manually that it should contain *one word per row*. Also, ensure that it has a header such as StopWords as Zingg ignores the header by default and works on the remaining data.
```
"fieldDefinition":[
{
"fieldName" : "fname",
"matchType" : "fuzzy",
"fields" : "fname",
"dataType": "string",
"stopWords": "models/100/stopWords/fname.csv"
},
```
In case of stopwords being set up manually by the user, the list of stopwords may consider multiple columns and Zingg used only the first column by default.
For recommending stopwords in **Zingg Enterprise Snowflake**,
`./scripts/zingg.sh --phase recommend --conf --properties-file --column `
The stopwords generated are stored in the table - zingg\_stopWords\_columnName\_modelId where we can see the list of stopwords and their frequency associated with the given column name.
# Defining Domain Specific Blocking And Similarity Functions
To add blocking functions and how they work
You can add your own [blocking functions](https://github.com/zinggAI/zingg/tree/main/common/core/src/main/java/zingg/common/core/hash) which will be evaluated by Zingg to build the [blocking tree.](/latest/zmodels)
The blocking tree works on the matched records provided by the user as part of the training. At every node, it selects the hash function and the field on which it should be applied so that there is the least elimination of the matching pairs.\
\
Say we have data like this:
| Pair 1 | firstname | lastname |
| :------: | :-------: | :------: |
| Record A | john | doe |
| Record B | johnh | d oe |
***
| Pair 2 | firstname | lastname |
| :------: | :-------: | :------: |
| Record A | mary | ann |
| Record B | marry | |
Let us assume we have hash function **first1char** and we want to check if it is a good function to apply to **firstname**:
| Pair | Record | Output |
| :--: | :------: | ------ |
| 1 | Record A | j |
| 1 | Record B | j |
| 2 | Record A | m |
| 2 | Record B | m |
There is no elimination in the pairs above, hence it is a good function.
Now let us try **last1char** on **firstname:**
| Pair | Record | Output |
| :--: | :------: | ------ |
| 1 | Record A | n |
| 1 | Record B | h |
| 2 | Record A | y |
| 2 | Record B | y |
Pair 1 is getting eliminated above, hence **last1char** is not a good function.
So, **first1char**(**firstname**) will be chosen. This brings near similar records together - in a way, clusters them to break the cartesian join.
These business-specific blocking functions go into [Hash Functions](https://github.com/zinggAI/zingg/tree/main/common/core/src/main/java/zingg/common/core/hash) and must be added to [HashFunctionRegistry](https://github.com/zinggAI/zingg/tree/0.7.0/common/core/src/main/java/zingg/common/core/hash/HashFunctionRegistry.java) and [hash functions config](https://github.com/zinggAI/zingg/tree/0.7.0/common/core/src/main/resources/hashFunctions.json).
Also, for similarity, you can define your own measures. Each **dataType** has predefined features, for example, [String](https://github.com/zinggAI/zingg/tree/0.7.0/common/core/src/main/java/zingg/common/core/feature/StringFeature.java) fuzzy type is configured for Affine and Jaro.
You can define your own [comparisons](https://github.com/zinggAI/zingg/tree/main/common/core/src/main/java/zingg/common/core/similarity/function) and use them.
# Interpreting Output Scores
For each field(**FNAME, NAME, EMAIL** ..), Zingg computes multiple features and feeds them to a classifier. These features are typically different ways to compare strings. We consider string lengths and their differences, character differences, and which characters *actually* differed.
* The shorter string pair ABCD and ABCE will be *less similar* than ABCDEF and ABCEEF.
* Common typos, for e.g. m instead of n, are penalized less severely and will be scored higher than a word with a replaced by non-obvious character b.
* The *more changes* you need to make to the strings to make them match, the less similar they will be.
* Differences in the middle are penalized more than prefixes and suffixes.
No individual feature is perfect, but the whole is greater than the sum of its parts. That's where the strength of Zingg matching and the accuracy comes from.
The classifier finds the best curve to represent the data(features) and gives a final score. This score is dependent on the individual features but is not a linear function but a curve fit. The deterioration in similarity does happen as scores get lower, but we try very hard to find as many possible matches, and hence you will continue to get good quality matches at lower scores. However, the number of good matches at lower scores will be lesser than those at higher scores.
The threshold is automatically chosen so that you can pick up most of the results, with a balance toward accuracy(what is a right match) and recall(not missing out on potential matches). Unlike traditional systems, we do not want the user to worry too much about the cut-off, We optimize the cutoff and hence you may see records with scores less than the conventional 0.5.
A few things to keep in mind while interpreting the scores:
* Matching is *transitive*, so if record A matches Record B and Record B matches C, we put records A, B, and C in the same cluster. That is how different records in a cluster get matched at various confidence levels to the rest and all these records showed up together.
* Our recommendation is to keep a threshold below **x**% of the max score as suspect records - manual review/to be visited later/flow through but will lessen confidence. **x** will depend on how accurate you find the results and how much you want to control the outcome.
* Keep cluster size above 4 or 5 for inspection. You could keep it irrespective of the score, or look at only those clusters whose **z\_minScore** is 0. This will depend on what the results look like to you.
# Explanation of Matches
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
Entity resolution is more effective when you can observe how clusters form, connect, and evolve across runs. Once you have run a `match` or `runIncremental` phase, you may like to look under the hood to understand how a cluster got formed and which records matched with each other and which did not. This insight helps with the following:
* Model Training and Validation: Quickly check if results match expectations.
* Traceability: See exactly how clusters are formed over time.
* Governance: Explain results to business stakeholders.
* Human-in-the-loop Reviews: Share pair-level evidence with domain experts.
* Operational Confidence: Build trust in production pipelines.
## Important Notes
The `explainOutput` phase currently focuses on **probabilistic matches** between records. Please note the following:
* **Deterministic matches** are not included in the explain output at this time. These matches are based on exact rules and can be reviewed separately through your match results.
* Some clusters may not generate explain output, particularly those formed primarily through deterministic matching rules.
* **Empty results** will occur if the specified `ZINGG_ID` does not exist in the cluster data. Please verify the cluster ID exists in your match output before running explain.
* For comprehensive cluster analysis, we recommend reviewing the primary match output alongside the explain output to get the complete picture of both deterministic and probabilistic relationships.
This focused approach allows the explain phase to efficiently highlight the machine learning model's decision-making process for probabilistic matches, which typically require the most scrutiny and validation.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Which records matched with each other in a cluster
Ready to audit a cluster? Plug in the Zingg ID and go. Get pair-level traceability for clusters.
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
Coming Soon!
Many times, records end up being in a cluster even when they may not have directly matched. For example Recod A matched with B, and B matched with C. Some records may match a lot more to other records in the cluster, while one record may just match with one andhence get included in the cluster. If you want to audit clusters, investigate anomalies, present rationale to stakeholders, or refine your matching strategy with confidence, it is important to understand pair wise matching.
The `explain` phase helps us learn this.
## Configuration
The `explain` configuration references the original matching config and defines where to write the results.
```json
{
"config": "path_to_original_matching_config/config.json",
"explainOutput": {
"name": "outputExplain",
"format": "csv",
"props": {
"location": "/tmp/zinggOutput_explain",
"delimiter": ",",
"header": true
}
}
}
```
Run the **explain** phase from your Zingg Enterprise Installation and pass the Zingg ID of the cluster you want to inspect:
```
./scripts/zingg.sh --phase explain --zinggid --conf
```
Example:
```
./scripts/zingg.sh --phase explain \
--zinggid ea67d79a-56a7-4431-ab55-d08bb3c10e2e \
--conf ./examples/febrl/explainConfig.json
```
This will provide a pair-level view `(pk1, pk2)` for the selected Zingg ID
* `pk1`: Primary key (record ID) of the first record in the pair
* `pk2`: Primary key (record ID) of the second record in the pair
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Output Statistics
Under the hoods of the matching process
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
If you’ve ever asked “how deterministic rules are performing?” or “did my latest incremental run improve cluster quality?”, Output Statistics is your answer. The Output Statistics surface information about the linkages Zingg found among records within a cluster. While running Zingg incrementally, Output Statistics expose how cluster numbers change as records get inserted and updated into the identity graph. Match Statistics surfaces those insights by writing structured metrics for every match or incremental run, so you can:
* See how dense or sparse your clusters are
* Understand how much of a cluster is explained by deterministic rules vs. probabilistic links
* Identify highly central records (connectors) and outliers
* Track how clusters change across runs (growth, splits, merges, reassignments)
If the number of clusters changes disproportionately to the number of records updated or added, an alert could be triggered.
***
## What gets written
Zingg writes statistics to the stats directory whenever you run phases like match or incremental. The output comprises of three types:
* SUMMARY: High-level run summary
* CLUSTER: One row per cluster with cluster level matching metrics
* RECORD: One row per record with its matching metrics within its cluster
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to [Zingg Entity Resolution Compare Versions](https://www.zingg.ai/product/zingg-entity-resolution-compare-versions) for individual tier features.
# Configuring Output Statistics
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
You can configure stats output either via JSON configuration or via the Python API. Please ensure the path/name contains the placeholder “*$ZINGG\_DYNAMIC\_STAT\_NAME*”. This placeholder will get substituted with the statistics type (SUMMARY, CLUSTER, RECORD) and written to respective locations.
### JSON configuration example
```json
{
"outputStats" : {
"name":"stats",
"format":"csv",
"props": {
"location": "/tmp/zinggStats_$ZINGG_DYNAMIC_STAT_NAME",
"delimiter": ",",
"header":true
}
}
}
```
### Python configuration example
```python
from zingg import ECsvPipe
statsOutputPipe = ECsvPipe("stats", "/tmp/febrlStats_$ZINGG_DYNAMIC_STAT_NAME")
statsOutputPipe.setHeader("true")
eArgs.setOutputStats(statsOutputPipe)
```
When you execute **match** or **incrementalRun** phases with the output stats pipe, Zingg writes stats for this run under the configured location with timestamp. If the outputStats are not configured, Zingg will not write them and the run will proceed as is.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Cluster Statistics
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
### Cluster Stats
* `z_cluster_count = n`: Number of records in the cluster (cluster size).
* `z_cluster_edges`: Total number of deterministic and probabilistic pairs among records in the cluster. This shows all the pairwise connections formed within the cluster.
* `z_cluster_deterministic_edges`: Total number of deterministic edges among records in the cluster as [configured through the determinstic rules](/latest/stepbystep/configuration/deterministicmatching).
* `z_cluster_centrality = z_cluster_edges / (nC2)`: Edge density of connections in the cluster. Values near 1 → dense clusters (almost all pairs connected); lower values → sparse clusters.
* `z_cluster_determinism = z_cluster_deterministic_edges / (nC2)`: Proportion of connectivity driven by deterministic rules. Higher means more rule-driven cluster cohesion.
Note:
* `nC2` (combinations of n records taken 2 at a time) equals `n*(n-1)/2`.
#### Tip
* Watch determinism ratio: Rising `z_cluster_determinism` over iterations usually means your rules are improving.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Record Statistics
Record level matching explanation
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
### Record Stats
* `z_record_deterministic_edges`: Number of records this record matches deterministically.
* `z_cluster_count = n`: Number of records in the cluster that this record is part of.
* `z_record_edges`: Total number of other records with which this record matches deterministically and probabilistically.
* `z_record_centrality = z_record_edges / (n - 1)`: How central the record is within its cluster.
* `z_record_determinism = z_record_deterministic_edges / (n - 1)`: Proportion of the record’s connectivity explained by deterministic rules.
#### Tip
* Audit connectors: A few records with very high `z_record_centrality` can sometimes falsely bridge clusters so validate them.
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Summary Statistics
Displays total number of records, clusters found after each match or incremental run with timestamp
# Combining Different Match Models
When a single match model is not sufficient
[Zingg Enterprise Feature](#user-content-fn-1)[^1]
In many cases, we want to build the identity graph using a combination of different datasets, schemas and matching logic. An example could be having a source system which only contains userids and emails, another one wtih user name and phone numbers and a few others with person information with addresses. Another example could be some systems capturing spousal information, but others to be matched on the basis of lastname and address.
In such cases, Zingg can build the entire graph and relate different models together. In the following case, results of a query with exact match on family Id and a matching model(household) using address and lastname are brought together.
````json
```
{
"vertices" :
[
{
"name" : "spouse",
"vertexType" : "zingg_pipe",
"data" : [
{
"name" : "spouse",
"format" : "snowflake",
"props": {
"query": "select a.id as id, a.FNAME, a.LNAME, a.STNO, a.ADD1, a.CITY, a.STATE, a.ZINGG_ID_PERSON, b.id as z_id, b.fname as Z_FNAME,b.lname as Z_LNAME,b.stno as Z_STNO,b.add1 as Z_ADD1, b.city as Z_CITY,b.state as Z_STATE, b.ZINGG_ID_PERSON as Z_ZINGG_ID_PERSON from CUSTOMER_RELATE_PARTIAL a, CUSTOMER_RELATE_PARTIAL b where a.familyId = b.familyId"
}
}
],
"edges" :
{ "edgeType" : "same_edge",
"edges":[
{
"dataColumn" : "zingg_personId",
"column" : "zingg_personId",
"name" : "zingg_personId1"
},
{
"dataColumn" : "zingg_personId",
"column" : "z_zingg_personId",
"name" : "zingg_personId2"
}
]
}
},
{
"name" : "household",
"config" : "$ZINGG_ENTERPRISE_HOME$/zinggEnterprise/configHousehold.json",
"strategy" : {
"vDataStrategy" : "unique_edge",
"props" : {
"column" : "zingg_personId",
"edge" : "zingg_personId,z_zingg_personId"
}
},
"vertexType" : "zingg_match",
"edges" :
{ "edgeType" : "same_edge",
"edges":[
{
"dataColumn" : "zingg_personId",
"column" : "zingg_personId",
"name" : "zingg_personId1"
},
{
"dataColumn" : "zingg_personId",
"column" : "z_zingg_personId",
"name" : "zingg_personId2"
}
]
}
}
],
"output" : [{
"name":"relatedCustomers",
"format":"snowflake",
"props": {
"table": "RELATED_CUSTOMERS_PARTIAL"
}
}],
"strategy":"pairs_and_vertices"
}
```
````
[^1]: Zingg Enterprise is the suite of proprietary products licensed by Zingg. Please refer to for individual tier features.
# Reporting Bugs And Contributing
Want to report a bug or request a feature? Let us know on [Slack](https://join.slack.com/t/zinggai/shared_invite/zt-w7zlcnol-vEuqU9m~Q56kLLUVxRgpOA), or open an [issue](https://github.com/zinggAI/zingg/issues/new/choose).
Want to commit code? Please take a look at the [open issues](https://github.com/zinggAI/zingg/issues) or choose the ones marked [good first issue](https://github.com/zinggAI/zingg/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22+). Let us talk on [Slack](https://join.slack.com/t/zinggai/shared_invite/zt-w7zlcnol-vEuqU9m~Q56kLLUVxRgpOA) if you need help deciding what to fix or how.
As the sponsor of Zingg, Zingg Labs would like to ensure the long-term viability of Zingg and its community. For all code contributions to Zingg, we ask that contributors complete and sign a Contributor License Agreement. The Contributor License Agreement helps ensure everyone can enjoy Zingg with confidence that Zingg is here to stay.
Specifically, our Contributor License Agreement (CLA) grants the contributor and Zingg Labs joint copyright interest in contributed code. Further, it provides assurance from the contributor that contributions are original work that does not violate any third-party license agreement. The agreement between contributors and the Zingg project is explicit. This ensures that Zingg users can be confident in the legal status of the source code and their right to use it.
* [Individual Contributor License Agreement v1.0](https://forms.gle/wU46nGvTMM2wSzy5A)
The above agreement is based on the Contributor License Agreements published by the [Apache Software Foundation](http://www.apache.org), with modifications:
* [Individual Contributor License Agreement ("Agreement") V2.0](http://apache.org/licenses/icla.pdf)
# Setting Up Zingg Development Environment
Welcome to the Zingg development setup guide. To provide you with the most accurate instructions, please select your operating system:
* [**macOS Setup Guide**](/latest/contributing/settingupzingg/macossetup)
* [**Ubuntu / WSL2 Setup Guide**](/latest/contributing/settingupzingg/ubuntusetup)
***
If you're using a different Linux distribution, the Ubuntu instructions will likely be the most relevant, but you may need to adjust package management commands (e.g., using `yum` or `dnf` instead of `apt`).
# macOS Setup Guide
This guide provides specific instructions for setting up the Zingg development environment on macOS.
### **Step 0: Initial OS Setup**
Homebrew is the preferred package manager for macOS and makes installing system dependencies much easier. While you can install dependencies manually, this guide uses Homebrew for simplicity.
Install Homebrew:
```bash
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
```
**Note for Apple Silicon:** If `brew` is not found, run: `echo 'eval "$(/opt/homebrew/bin/brew shellenv)"' >> ~/.zshrc && source ~/.zshrc`
***
### **Step 1: Clone The Zingg Repository**
```bash
brew install git
```
**Note:** It is suggested to fork the repository to your account and then clone the repository.
```bash
git clone https://github.com/zinggAI/zingg.git
```
***
### **Step 2: Install JDK 11 (Java Development Kit)**
* Install OpenJDK 11: `brew install openjdk@11`
* Link Java 11:
* **Apple Silicon:** `sudo ln -sfn /opt/homebrew/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk`
* **Intel:** `sudo ln -sfn /usr/local/opt/openjdk@11/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-11.jdk`
***
### **Step 3: Install Apache Spark**
* Download Apache Spark from the [Apache Spark Official Website](https://spark.apache.org/downloads.html).
* For example for 3.5.0:
```bash
curl -O https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark
rm spark-3.5.0-bin-hadoop3.tgz
```
Make sure that Spark version you have installed is compatible with Java you have installed, and Zingg is supporting those versions.
**Note**: Zingg supports Spark 3.5 and the corresponding Java version.
***
### **Step 4: Install Apache Maven**
```bash
brew install maven
```
***
### **Step 5: Update Environment Variables (\~/.zshrc)**
```bash
export JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk-11.jdk/Contents/Home
export SPARK_HOME=/opt/spark
export SPARK_MASTER=local[*]
export ZINGG_HOME=/assembly/target
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin
```
`` will be a directory where you clone the repository of the Zingg. Similarly, if you have installed spark on a different directory you can set **SPARK\_HOME** accordingly.
* Save/exit and do source .zshrc so that they reflect:
```bash
source ~/.zshrc
```
* Verify:
```bash
echo $PATH
mvn --version
```
***
### **Step 6: Compile The Zingg Repository**
* Make sure you are executing the following commands in the same terminal window where you saved the .zshrc. Run the following to compile the Zingg Repository:
```bash
git branch
```
* Run the following to Compile the Zingg Repository:
```bash
mvn initialize
mvn clean compile package -Dspark=sparkVer
```
* Run the following to Compile while skipping tests:
```bash
mvn initialize
mvn clean compile package -Dspark=sparkVer -Dmaven.test.skip=true
```
**Note:** Replace the `sparkVer` with the version of Spark you installed. For example, **-Dspark=3.5**. If you still face an error, include **-Dmaven.test.skip=true** with the above command.
***
### **Step 7: If you have any issue with 'SPARK\_LOCAL\_IP'**
1. **Find your IP:** `ipconfig getifaddr en0`
2. **Find your Hostname:** `hostname`
3. **Update hosts:** `sudo nano /etc/hosts` and add `[your-ip] [your-hostname]` at the bottom.
***
### **Step 8: Run Zingg To Find Training Data**
* Run this script in the terminal opened in Zingg clones directory:
```bash
./scripts/zingg.sh --phase findTrainingData --conf examples/febrl/config.json
```
**If everything is right, it should show Zingg banner.**
***
### **Step 9: Run Zingg To Label Data**
* Run this script in the terminal opened in Zingg clones directory:
```bash
./scripts/zingg.sh --phase label --conf examples/febrl/config.json --properties-file config/zingg.conf
```
***
### **Step 10: Run Zingg To Train Model Based On Labeling**
* Run this script in the terminal opened in Zingg clones directory:
```bash
./scripts/zingg.sh --phase train --conf examples/febrl/config.json --properties-file config/zingg.conf
```
***
### **Step 11: Run Zingg To Prepare Final Output Data**
* Run this script in the terminal opened in Zingg clones directory:
```bash
./scripts/zingg.sh --phase match --conf examples/febrl/config.json --properties-file config/zingg.conf
```
* Change directory `cd /tmp/zinggOutput` (or the path provided in your config file) to see the output files.
# Ubuntu/WSL2 Setup Guide
The following steps will help you set up the Zingg Development Environment on **Ubuntu/WSL2**.
### **Step 0: Initial OS Setup (Ubuntu/WSL2)**
Make sure to update your Ubuntu installation:
`sudo apt update`
**Step 0: Install Ubuntu on WSL2 on Windows**
* Install **wsl**: Type the following command in **Windows PowerShell**.
```
wsl --install
```
* Download Ubuntu from **Microsoft Store**, **Ubuntu 20.04 LTS**
* Configure Ubuntu with a **username** and **password**
* Open **Ubuntu 20.04 LTS** and start working
```
sudo apt update
```
* Follow this [tutorial](https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-10#1-overview) for more information.
**Step 1: Clone The Zingg Repository (Ubuntu)**
* Install and SetUp Git: **sudo apt install git**
* Verify : **git --version**
* Set up Git by following the [tutorial](https://www.digitalocean.com/community/tutorials/how-to-install-git-on-ubuntu-20-04).
* Clone the Zingg Repository: **git clone **
**Note:** It is suggested to fork the repository to your account and then clone the repository.
**Step 2: Install JDK 11 (Ubuntu)**
* Follow this [tutorial](https://linuxize.com/post/install-java-on-ubuntu-20-04/) to install Java 11 JDK 11 in Ubuntu.
* For example:
```
sudo apt install openjdk-11-jdk openjdk-11-jre
javac -version
java -version
```
**Step 3: Install Apache Spark**
#### **Common Steps**
* Download Apache Spark - from the [Apache Spark Official Website](https://spark.apache.org/downloads.html).
* For example for 3.5.0:
```bash
curl -O https://archive.apache.org/dist/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark
rm spark-3.5.0-bin-hadoop3.tgz
```
#### **Original Ubuntu Instructions (Manual Wget)**
* Install downloaded Apache Spark - on your Ubuntu by following [this tutorial](https://computingforgeeks.com/how-to-install-apache-spark-on-ubuntu-debian/).
* For example for 3.5.0:
```
wget https://www.apache.org/dyn/closer.lua/spark/spark-3.5.0/spark-3.5.0-bin-hadoop3.tgz
tar -xvf spark-3.5.0-bin-hadoop3.tgz
rm -rf spark-3.5.0-bin-hadoop3.tgz
sudo mv spark-3.5.0-bin-hadoop3 /opt/spark
```
Make sure that Spark version you have installed is compatible with Java you have installed, and Zingg is supporting those versions.
**Note**: Zingg supports Spark 3.5 and the corresponding Java version.
**Step 4: Install Apache Maven (Ubuntu)**
* Install the latest **maven** package.
* For example for 3.8.8:
```
wget https://dlcdn.apache.org/maven/maven-3/3.8.8/binaries/apache-maven-3.8.8-bin.tar.gz
tar -xvf apache-maven-3.8.8-bin.tar.gz
rm -rf apache-maven-3.8.8-bin.tar.gz
cd apache-maven-3.8.8/
cd bin
./mvn --version
Make sure that mvn -version should display correct java version as well(JAVA 11)
Apache Maven 3.8.7
Maven home: /usr/share/maven
Java version: 11.0.23, vendor: Ubuntu, runtime: /usr/lib/jvm/java-11-openjdk-amd64
```
**Step 5: Update Environment Variables (Ubuntu - \~/.bashrc)**
Open `.bashrc` and add env variables at the end of the file.
```
vim ~/.bashrc
export SPARK_HOME=/opt/spark
export SPARK_MASTER=local[*]
export MAVEN_HOME=/home/ubuntu/apache-maven-3.8.8
export ZINGG_HOME=/assembly/target
export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin:$JAVA_HOME/bin
```
\ will be a directory where you clone the repository of the Zingg. Similarly, if you have installed spark on a different directory you can set **SPARK\_HOME** accordingly.
**Note :-** Skip exporting MAVEN\_HOME if multiple maven version are not required
* Save/exit and do source .bashrc so that they reflect
```
source ~/.bashrc
```
* Verify:
```
echo $PATH
mvn --version
```
**Note:** If you have already set up **JAVA\_HOME** and **SPARK\_HOME** in the steps before you don't need to do this again.
**Step 6: Compile The Zingg Repository**
* Make sure you are executing the following commands in the same terminal window where you saved the bashrc. Run the following to compile the Zingg Repository -
```
git branch
```
* Run the following to Compile the Zingg Repository
```
mvn initialize
mvn clean compile package -Dspark=sparkVer
```
* Run the following to Compile while skipping tests
```
mvn initialize
mvn clean compile package -Dspark=sparkVer -Dmaven.test.skip=true
```
**Note:** Replace the `sparkVer` with the version of Spark you installed.\
For example, **-Dspark=3.5** you still face an error, include **-Dmaven.test.skip=true** with the above command.
**Step 7: If you have any issue with 'SPARK\_LOCAL\_IP' (Ubuntu)**
* Install **net-tools** using **sudo apt-get install -y net-tools**
* Run `ifconfig` in the terminal, find the **IP address** and paste the same in **/opt/hosts** IP address of your Pc-Name
**Step 8: Run Zingg To Find Training Data**
* Run this script in the terminal opened in Zingg clones directory `./scripts/zingg.sh --phase findTrainingData --conf examples/febrl/config.json`
**If everything is right, it should show Zingg banner.**
**Step 9: Run Zingg To label Data**
* Run this script in the terminal opened in Zingg clones directory `./scripts/zingg.sh --phase label --conf examples/febrl/config.json --properties-file config/zingg.conf`
**Step 10: Run Zingg To train model based on labelling**
* Run this script in the terminal opened in Zingg clones directory `./scripts/zingg.sh --phase train --conf examples/febrl/config.json --properties-file config/zingg.conf`
**Step 11: Run Zingg To prepare final output data**
* Run this script in the terminal opened in Zingg clones directory `./scripts/zingg.sh --phase match --conf examples/febrl/config.json --properties-file config/zingg.conf`
* change directory `cd /tmp/zinggOutput` (path provided in config file) to see the output files.
# Community
Be part of the conversation in the [Zingg Community Slack](https://join.slack.com/t/zinggai/shared_invite/zt-w7zlcnol-vEuqU9m~Q56kLLUVxRgpOA).
# Frequently Asked Questions
## How much training is enough?
Typically 30-40 positive pairs (matches) should build a good first model. While marking records through the interactive learner, you can check Zingg's predictions for the shown pair. If they seem to be correct, you can pause and run Zingg in train and match phases to see what result you are getting. If not satisfied, you can always run the findTrainingData and label jobs again and they will pick from the last training round.
## Do I need to train for every new dataset?
No, absolutely not! Train only if the schema(attributes or their types) has changed.
## Do I need to use a Spark cluster or can I run on a single machine?
Depends on the data size you have. Check [hardware sizing](/latest/stepbystep/hardwaresizing) for more details.
## I dont have much background in ML or Spark. Can I still use Zingg ?
Very much! Zingg uses Spark and ML under the hood so that you don't have to worry about the rules and the scale.
## Is Zingg an MDM?
Zingg Community Version is a DIY Master Data Management Product, solving the entity matching and linking process. You can build an MDM in a data store of your choice using Zingg Community. Zingg Enterprise Version is a lakehouse/warehouse native MDM with advance features for globally unique and peristent identifier, gden records, agentic stewardship and survivorship.
## Is Zingg a CDP ?
No, Zingg is not a CDP. However, Zingg is part of the Composable CDP stack, where enterprises assemble customer platform off your warehouse or datalake, Zingg is a great fit. You can leverage existing ETL, observability, and other tools which are part of your data stack and use Zingg for identity.\
Zingg Community Version can be used to build a composable CDP by identity resolution natively on the warehouse and datalake and building customer 360 views. Zingg's identity resolution is far more powerful than what is offered by any out of the box CDP.\
Zingg Enterprise's probabilistic and deterministic matching take this further beyond. Here is an [article](https://hightouch.com/blog/warehouse-identity-resolution/) describing how you can build your own CDP on the warehouse with Zingg.
## I can do Entity Resolution using a graph database like TigerGraph/Neo4J, why do I need Zingg ?
Doing entity resolution in graph databases is easy only if you have trusted and high-quality identifiers like passport id, SSN id, etc. through which edges can be defined between different records. If you need fuzzy matching, you will have to build your own rules and algorithms with thresholds to define matching records. Zingg and Graph Databases go hand in hand for Entity Resolution. It is far easier to use Zingg and persist its graph output to a graph database and do further processing for AML, and KYC scenarios there. Read [the article](https://towardsdatascience.com/entity-resolution-with-tigergraph-add-zingg-to-the-mix-95009471ca02) for details on how Zingg uses TigerGraph for Entity Resolution.
# Reading Material
Entity Resolution and The Modern Data Stack:
* [From Rows to People](https://roundup.getdbt.com/p/from-rows-to-people)
Identity Resolution and Why CDPs fail:
* [Customer Journey](https://www.gartner.com/en/articles/customer-journey)
Entity Resolution using a graph database:
* [Entity Resolution with TigerGraph](https://towardsdatascience.com/entity-resolution-with-tigergraph-add-zingg-to-the-mix-95009471ca02)
A detailed write-up on entity resolution - the problem, its challenges, and applications:
* [Entity Resolution](https://towardsdatascience.com/an-introduction-to-entity-resolution-needs-and-challenges-97fba052dde5)
Understanding Master Data and Master Data Management:
* [Agile Data Mastering](https://towardsdatascience.com/a-guide-to-agile-data-mastering-with-ai-3bf38f103709)
# Security And Privacy
A note about telemetry and product usage data collection in Zingg
Zingg models are built on your data and deployed within your network. No data leaves your environment.
However, Zingg does collect usage metrics and writes them to Google Analytics. This is done to understand and improve the user experience. Please be assured that **Zingg does not capture any input data and will never do so. No input or output data ever leaves any Zingg deployment. The privacy and governance settings of the execution eivironments of Zingg are always enforced due to the native and on-premise deployment of all editions and versions of Zingg. This is valid for the base Operating System, Databricks, Snowflake, Bigquery, Fabirc, Azure, AWS, GCP, Private Data Centre or any other platforms where you run Zingg.**
The following details are captured:
* **Data source type:** data format e.g. CSV, snowflake, parquet
* **Fields count:** number of fields used for training
* **Record Count:** total number of total records pasing through Zingg
* **Execution Time:** execution time of the program
* **Matches and Nonmatches:** the number of matched and nonmatched records in the model
* **Domain:** helps Zingg developers to understand our user enterprises
* **Zingg user id:** Zingg OS User ID
* **JDK information:** version, flavour
* **OS information:** operating system type
* **Databricks Environment:**
An example analytics event while running Zingg is
```
event is {"client_id":"localhost","events":[{"name":"match","params":{"executionTime":"1.743246748E9","country":"IN","zingg_version":"0.5.0","modelId":"100","domain":"localhost","java_version":"11.0.18","os_arch":"aarch64","os_name":"Mac OS X","DB_INSTANCE_TYPE":null,"DATABRICKS_RUNTIME_VERSION":null,"dataCount":"65.0","ZINGG_HOME":"true"}}],"user_id":"zingg"}
```
To view the information captured while running Zingg, edit log4j2.properties and turn loggin level of zingg\_analytics to warn
```
logger.zingg_analytics.level = warn
```
If you do not wish to send the data, please set collectMetrics flag to false in the configuration JSON while running Zingg. A blank event will be logged in this case
```
{"client_id":"localhost","events":[{"name":"train"}],"user_id":"zingg"}
```
