Platform Guide for GCP Dataproc
End-to-end guide to running Zingg on Google Cloud Dataproc from GCS storage setup and cluster creation to your first match result. Covers Community (open source) and Enterprise.
Combining Zingg with Google Cloud gives you elastic Spark scale via Dataproc, flexible storage via GCS, and a managed JupyterLab workspace via the Component Gateway. Spin up a cluster when you need it and shut it down when the job is done.
Before you start
Each item below blocks cluster creation or notebook setup, and the resulting error messages are not self-explanatory.
The Compute Engine default service account (
[email protected]) needsroles/dataproc.workerandroles/storage.objectAdmin. On newer projects these are not granted automatically.You need
roles/dataproc.editorto create and manage clusters, androles/iap.tunnelResourceAccessorif you want SSH access to the master node.The Cloud Dataproc, Compute Engine, and Cloud Storage APIs must be enabled.
Your Dataproc cluster has no outbound internet access by default. Every Python package must be installed from a wheel staged in GCS —
pip installfrom PyPI fails withNetwork is unreachable.
To grant the service account roles:
PROJECT=$(gcloud config get-value project)
SA="$(gcloud projects describe $PROJECT --format='value(projectNumber)')[email protected]"
gcloud projects add-iam-policy-binding $PROJECT \
--member="serviceAccount:$SA" --role="roles/dataproc.worker"
gcloud projects add-iam-policy-binding $PROJECT \
--member="serviceAccount:$SA" --role="roles/storage.objectAdmin"Step 1: Download JARs and prepare your GCS bucket
Enterprise requires four JARs — the Enterprise engine, your license, and the two Google connectors.
zingg-enterprise-spark-1.0.0.jar
The Enterprise engine
Provided with your Enterprise distribution
zingg_license.jar
Your Enterprise license
Provided with your Enterprise distribution
spark-3.5-bigquery-0.44.1.jar
BigQuery connector
github.com/GoogleCloudDataproc/spark-bigquery-connector
gcs-connector-hadoop3-latest.jar
GCS connector
docs.cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage
You also need three Python wheels — zinggEC and zinggES from your Enterprise distribution, plus the open-source zingg package that they extend.
Community requires three JARs — the Zingg engine and the two Google connectors.
zingg-0.6.0.jar
The Zingg engine
github.com/zinggAI/zingg/releases
spark-3.5-bigquery-0.44.1.jar
BigQuery connector
github.com/GoogleCloudDataproc/spark-bigquery-connector
gcs-connector-hadoop3-latest.jar
GCS connector
docs.cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage
No license JAR and no Python wheels are needed — the zingg package installs from PyPI.
Create a GCS bucket in the region where your Dataproc cluster will run, then upload the JARs, any wheels, and your dataset. Use either the Cloud Console or the gcloud CLI.
Cloud Console
Navigate to Cloud Storage → Buckets → Create.
Name your bucket with a globally unique identifier (for example
zingg-production-storage).Set Location type to Region and pick the region your Dataproc cluster will run in (for example
us-central1).Under Choose how to control access, select Uniform — this matches
--uniform-bucket-level-accessin the CLI form below.Click Create.
Open the bucket and click UPLOAD FILES. Upload the JARs from the table above, plus the Python wheels if you are on Enterprise.
Click CREATE FOLDER, name it
data, open it, and upload your dataset (for exampletest.csv). Step 5 reads fromgs://YOUR_BUCKET/data/.Note the exact filenames in the bucket listing — Step 2 needs them verbatim.

gcloud CLI
Step 2: Create a Dataproc cluster
All four JARs must be injected via spark.jars at creation time. This cannot be changed on a running cluster.
gcloud CLI
For a single-node cluster, replace the three machine flags with:
Verify the JARs registered. If this prints fewer than four paths, delete and recreate:
Cloud Console
Search for Managed Apache Spark (formerly Dataproc), go to Clusters, and select Create Cluster on Compute Engine.
Under Optional components, check Jupyter Notebook.
Under Component Gateway, check Enable component gateway.
Set the node machine type to
n2-standard-8with a 100GB boot disk.Under Customize cluster → Cluster properties, click + ADD PROPERTIES:
Prefix:
sparkKey:
spark.jarsValue: the four
gs://paths, comma-separated with no spaces
Add the remaining properties, each with the same
sparkprefix:spark.driver.memory=12gspark.executor.memory=8gspark.driver.maxResultSize=4gspark.eventLog.enabled=falsespark.sql.maxPlanStringLength=8192spark.sql.shuffle.partitions=8
Under Scheduled deletion, check Delete on idle and set 2 hours.
Under Manage security, check Enables the cloud-platform scope for this cluster.
Step 3: Open JupyterLab
Once the cluster reports Running, access JupyterLab through the Component Gateway — no SSH or firewall configuration needed.

Navigate to Dataproc → Clusters.
Click your cluster name.
Click the Web Interfaces tab.
Under Component Gateway, click the JupyterLab link.
Create a new notebook and select the PySpark kernel.


In JupyterLab, set the kernel in two clicks:
Click the kernel indicator at the top right of the notebook toolbar — marked 1 below. It reads
No Kerneluntil one is attached, and clicking it opens the Select Kernel dialog.Choose PySpark — marked 2 — not Python 3, then confirm.
The same indicator doubles as your check: after selecting, it should read PySpark before you run any cell.

Or retrieve the link from the CLI:
Step 4: Install the Python packages
Enterprise ships two wheels — zinggEC (common) and zinggES (Spark). Both subclass the open-source zingg package, so all three must be installed.
Restart the kernel (Kernel → Restart Kernel), then confirm all three are present:
You should see zingg, zinggEC, and zinggES.

Community needs only the open-source zingg package.
Restart the kernel (Kernel → Restart Kernel), then confirm it is present:
You should see zingg.
Step 5: Set paths and a checkpoint directory
modelId is a unique name for this model run — Zingg uses it as the folder name under zinggDir. Use the same values across every step.
The checkpoint directory is not optional. Zingg's iterative blocking builds a deeply nested Spark query plan, and checkpointing truncates that lineage between iterations. Without it the plan grows until the driver runs out of heap.
Step 6: Import libraries
Step 7: Build the arguments object
EArguments is the central configuration object. Every phase reads from the same instance.
Arguments()
EArguments()
FieldDefinition(...)
EFieldDefinition(...)
CsvPipe(...)
ECsvPipe(...)
ZinggWithSpark(...)
EZinggWithSpark(...)
ClientOptions(...)
ClientOptions(...) — unchanged
Enterprise also provides DeterministicMatching, InMemoryPipe, and UCPipe.
Step 8: Preview your data
Check whether your file has a header row before reading it. The FEBRL sample data ships without one:
Step 9: Configure input and output pipes
ECsvPipe connects Zingg to your GCS data. The schema string must match your column names and order exactly.
Step 10: Define fields and match types
Every field in your schema must appear in fieldDefinition. List the most important fields first — field order affects blocking quality.
Step 11: Configure performance settings
For a small dataset on a single-node cluster, use 2–4 partitions. For a multi-worker cluster, use roughly 20–30× the total worker vCPU count.
Step 12: Find candidate pairs
Zingg scans your dataset and identifies pairs the model is uncertain about — the edge cases where human input is most valuable. Candidate pairs are written to UNMARKED_DIR.
Step 13: Load pairs for labeling
Step 14: Label pairs in the widget
A widget displays each candidate pair side by side. For each pair select Match, No Match, or Uncertain.
Step 15: Save labeled pairs
Step 16: Train and match
trainMatch combines the train and match phases into a single call. Zingg builds a model from your labeled pairs and immediately applies it to the full dataset. This is the most compute-intensive step.
Step 17: View output
Match output is written to output_path as distributed part-files. Inspect the actual columns before renaming them — the score columns Zingg emits vary by version:
Then apply names matching the column count you saw:
Collapse duplicates by grouping on the cluster ID:
Troubleshooting
Default Service Account ... is missing required permissions: [dataproc.agents.create, ...]
The Compute Engine default service account lacks roles/dataproc.worker. Grant it along with roles/storage.objectAdmin, then wait 60 seconds for IAM to propagate.
UNAVAILABLE ... zone does not have enough resources
Zone capacity shortage. Omit --zone so Dataproc retries across zones, try e2-standard-8 instead of n2-standard-8, or switch region.
ALREADY_EXISTS: Failed to create cluster
A previous failed attempt left the cluster in ERROR state. Delete it before retrying.
Console shows only Cluster on GKE
Use the gcloud CLI. Creation works through the API regardless of what the console displays.
HTTP 502 on the JupyterLab link
Jupyter has not finished starting — wait 2–3 minutes after RUNNING. If it persists after a stop/start cycle, delete and recreate the cluster.
Network is unreachable during pip install
The cluster has no outbound internet. Stage the wheel in GCS and install from the local file.
ModuleNotFoundError: No module named 'zingg'
The open-source base package is missing, or the kernel was not restarted after installing. Install all three wheels, then Kernel → Restart Kernel.
NameError: name 'ECsvPipe' is not defined
Missing import. Each Enterprise class comes from its own module — see Step 6.
Cannot call methods on a stopped SparkContext
Two phases were run in separate cells, each creating a new context. Use trainMatch, or restart the kernel between phases.
ConnectionRefusedError: [Errno 111]
The JVM is gone — usually a consequence of the error above, or a driver OOM. Restart the kernel and re-run from Step 5.
java.lang.OutOfMemoryError: Java heap space
Driver heap exhausted building the query plan. Recreate on a *-standard-8 node with spark.driver.memory=12g and confirm the checkpoint directory is set.
ClassNotFoundException on the first Zingg call
A path in spark.jars is wrong. The cluster builds fine regardless — verify paths against gcloud storage ls.
Last updated
Was this helpful?