Building a continuosly updated identity graph with new, updated and deleted records
Rerunning matching on entire datasets is wasteful, and we lose the lineage of matched records against a persistent identifier. Using the incremental flow feature in Zingg Enterprise, incremental loads can be run to match existing pre-resolved entities. The new and updated records are matched to existing clusters, and new persistent ZINGG_IDs are generated for records that do not find a match. If a record gets updated and Zingg Enterprise discovers that it is a more suitable match with another cluster, it will be reassigned. Cluster assignment, merge, and unmerge happens automatically in the flow. Zingg Enterprise also takes care of human feedback on previously matched data to ensure that it does not override the approved records.
The incremental phase is run as follows:
./scripts/zingg.sh --phase runIncremental --conf <location to incrementalConf.json>