Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Platform Guide for GCP Dataproc

End-to-end guide to running Zingg on Google Cloud Dataproc from GCS storage setup and cluster creation to your first match result. Covers Community (open source) and Enterprise.

Combining Zingg with Google Cloud gives you elastic Spark scale via Dataproc, flexible storage via GCS, and a managed JupyterLab workspace via the Component Gateway. Spin up a cluster when you need it and shut it down when the job is done.

The steps below use the Enterprise API - EArguments, EFieldDefinition, ECsvPipe, and EZinggWithSpark. Where Community differs, both versions are shown in a tab. Everything untabbed applies to both editions.

Before you start

Each item below blocks cluster creation or notebook setup, and the resulting error messages are not self-explanatory.

  • The Compute Engine default service account ([email protected]) needs roles/dataproc.worker and roles/storage.objectAdmin. On newer projects these are not granted automatically.

  • You need roles/dataproc.editor to create and manage clusters, and roles/iap.tunnelResourceAccessor if you want SSH access to the master node.

  • The Cloud Dataproc, Compute Engine, and Cloud Storage APIs must be enabled.

  • Your Dataproc cluster has no outbound internet access by default. Every Python package must be installed from a wheel staged in GCS — pip install from PyPI fails with Network is unreachable.

To grant the service account roles:

PROJECT=$(gcloud config get-value project)
SA="$(gcloud projects describe $PROJECT --format='value(projectNumber)')[email protected]"

gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:$SA" --role="roles/dataproc.worker"

gcloud projects add-iam-policy-binding $PROJECT \
  --member="serviceAccount:$SA" --role="roles/storage.objectAdmin"

Step 1: Download JARs and prepare your GCS bucket

Enterprise requires four JARs — the Enterprise engine, your license, and the two Google connectors.

JAR
Purpose
Source

zingg-enterprise-spark-1.0.0.jar

The Enterprise engine

Provided with your Enterprise distribution

zingg_license.jar

Your Enterprise license

Provided with your Enterprise distribution

spark-3.5-bigquery-0.44.1.jar

BigQuery connector

github.com/GoogleCloudDataproc/spark-bigquery-connector

gcs-connector-hadoop3-latest.jar

GCS connector

docs.cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage

You also need three Python wheels — zinggEC and zinggES from your Enterprise distribution, plus the open-source zingg package that they extend.

Community requires three JARs — the Zingg engine and the two Google connectors.

JAR
Purpose
Source

zingg-0.6.0.jar

The Zingg engine

github.com/zinggAI/zingg/releases

spark-3.5-bigquery-0.44.1.jar

BigQuery connector

github.com/GoogleCloudDataproc/spark-bigquery-connector

gcs-connector-hadoop3-latest.jar

GCS connector

docs.cloud.google.com/dataproc/docs/concepts/connectors/cloud-storage

No license JAR and no Python wheels are needed — the zingg package installs from PyPI.

Create a GCS bucket in the region where your Dataproc cluster will run, then upload the JARs, any wheels, and your dataset. Use either the Cloud Console or the gcloud CLI.

Cloud Console

  1. Navigate to Cloud Storage → Buckets → Create.

  2. Name your bucket with a globally unique identifier (for example zingg-production-storage).

  3. Set Location type to Region and pick the region your Dataproc cluster will run in (for example us-central1).

  4. Under Choose how to control access, select Uniform — this matches --uniform-bucket-level-access in the CLI form below.

  5. Click Create.

  6. Open the bucket and click UPLOAD FILES. Upload the JARs from the table above, plus the Python wheels if you are on Enterprise.

  7. Click CREATE FOLDER, name it data, open it, and upload your dataset (for example test.csv). Step 5 reads from gs://YOUR_BUCKET/data/.

  8. Note the exact filenames in the bucket listing — Step 2 needs them verbatim.

GCS bucket Objects tab in the Google Cloud Console with the Upload menu open, showing Upload files and Upload folder alongside the Create folder button.

gcloud CLI

Step 2: Create a Dataproc cluster

All four JARs must be injected via spark.jars at creation time. This cannot be changed on a running cluster.

gcloud CLI

For a single-node cluster, replace the three machine flags with:

Verify the JARs registered. If this prints fewer than four paths, delete and recreate:

Cloud Console

  1. Search for Managed Apache Spark (formerly Dataproc), go to Clusters, and select Create Cluster on Compute Engine.

  2. Under Optional components, check Jupyter Notebook.

  3. Under Component Gateway, check Enable component gateway.

  4. Set the node machine type to n2-standard-8 with a 100GB boot disk.

  5. Under Customize cluster → Cluster properties, click + ADD PROPERTIES:

    • Prefix: spark

    • Key: spark.jars

    • Value: the four gs:// paths, comma-separated with no spaces

  6. Add the remaining properties, each with the same spark prefix:

    • spark.driver.memory = 12g

    • spark.executor.memory = 8g

    • spark.driver.maxResultSize = 4g

    • spark.eventLog.enabled = false

    • spark.sql.maxPlanStringLength = 8192

    • spark.sql.shuffle.partitions = 8

  7. Under Scheduled deletion, check Delete on idle and set 2 hours.

  8. Under Manage security, check Enables the cloud-platform scope for this cluster.

Step 3: Open JupyterLab

Once the cluster reports Running, access JupyterLab through the Component Gateway — no SSH or firewall configuration needed.

Dataproc Clusters list with zingg-cluster in us-central1 and its status circled, reading Running.
  1. Navigate to Dataproc → Clusters.

  2. Click your cluster name.

  3. Click the Web Interfaces tab.

  4. Under Component Gateway, click the JupyterLab link.

  5. Create a new notebook and select the PySpark kernel.

Dataproc Cluster details page for zingg-cluster with Status showing Running, and the Web Interfaces tab circled at the end of the tab row below the details table.
The Web Interfaces tab open, with the Component gateway section listing YARN ResourceManager, Spark History Server and Jupyter, and the JupyterLab link circled at the bottom.

In JupyterLab, set the kernel in two clicks:

  1. Click the kernel indicator at the top right of the notebook toolbar — marked 1 below. It reads No Kernel until one is attached, and clicking it opens the Select Kernel dialog.

  2. Choose PySpark — marked 2 — not Python 3, then confirm.

The same indicator doubles as your check: after selecting, it should read PySpark before you run any cell.

JupyterLab with the Select Kernel dialog open. Marker 1 circles the kernel indicator reading No Kernel at the top right of the toolbar; marker 2 circles the ticked PySpark entry in the dialog, above Python 3.

Or retrieve the link from the CLI:

Step 4: Install the Python packages

Enterprise ships two wheels — zinggEC (common) and zinggES (Spark). Both subclass the open-source zingg package, so all three must be installed.

Restart the kernel (Kernel → Restart Kernel), then confirm all three are present:

You should see zingg, zinggEC, and zinggES.

Notebook cell running pip list piped to grep zingg, with zingg 0.6.0, zinggEC 1.0.0 and zinggES 1.0.0 boxed in the output.

Community needs only the open-source zingg package.

Restart the kernel (Kernel → Restart Kernel), then confirm it is present:

You should see zingg.

Step 5: Set paths and a checkpoint directory

modelId is a unique name for this model run — Zingg uses it as the folder name under zinggDir. Use the same values across every step.

The checkpoint directory is not optional. Zingg's iterative blocking builds a deeply nested Spark query plan, and checkpointing truncates that lineage between iterations. Without it the plan grows until the driver runs out of heap.

Step 6: Import libraries

Step 7: Build the arguments object

EArguments is the central configuration object. Every phase reads from the same instance.

Community class
Enterprise equivalent

Arguments()

EArguments()

FieldDefinition(...)

EFieldDefinition(...)

CsvPipe(...)

ECsvPipe(...)

ZinggWithSpark(...)

EZinggWithSpark(...)

ClientOptions(...)

ClientOptions(...) — unchanged

Enterprise also provides DeterministicMatching, InMemoryPipe, and UCPipe.

Step 8: Preview your data

Check whether your file has a header row before reading it. The FEBRL sample data ships without one:

Step 9: Configure input and output pipes

ECsvPipe connects Zingg to your GCS data. The schema string must match your column names and order exactly.

Step 10: Define fields and match types

Every field in your schema must appear in fieldDefinition. List the most important fields first — field order affects blocking quality.

Step 11: Configure performance settings

For a small dataset on a single-node cluster, use 2–4 partitions. For a multi-worker cluster, use roughly 20–30× the total worker vCPU count.

Step 12: Find candidate pairs

Zingg scans your dataset and identifies pairs the model is uncertain about — the edge cases where human input is most valuable. Candidate pairs are written to UNMARKED_DIR.

Step 13: Load pairs for labeling

Step 14: Label pairs in the widget

A widget displays each candidate pair side by side. For each pair select Match, No Match, or Uncertain.

Step 15: Save labeled pairs

Step 16: Train and match

trainMatch combines the train and match phases into a single call. Zingg builds a model from your labeled pairs and immediately applies it to the full dataset. This is the most compute-intensive step.

Step 17: View output

Match output is written to output_path as distributed part-files. Inspect the actual columns before renaming them — the score columns Zingg emits vary by version:

Then apply names matching the column count you saw:

Collapse duplicates by grouping on the cluster ID:

Troubleshooting

Error
Cause and fix

Default Service Account ... is missing required permissions: [dataproc.agents.create, ...]

The Compute Engine default service account lacks roles/dataproc.worker. Grant it along with roles/storage.objectAdmin, then wait 60 seconds for IAM to propagate.

UNAVAILABLE ... zone does not have enough resources

Zone capacity shortage. Omit --zone so Dataproc retries across zones, try e2-standard-8 instead of n2-standard-8, or switch region.

ALREADY_EXISTS: Failed to create cluster

A previous failed attempt left the cluster in ERROR state. Delete it before retrying.

Console shows only Cluster on GKE

Use the gcloud CLI. Creation works through the API regardless of what the console displays.

HTTP 502 on the JupyterLab link

Jupyter has not finished starting — wait 2–3 minutes after RUNNING. If it persists after a stop/start cycle, delete and recreate the cluster.

Network is unreachable during pip install

The cluster has no outbound internet. Stage the wheel in GCS and install from the local file.

ModuleNotFoundError: No module named 'zingg'

The open-source base package is missing, or the kernel was not restarted after installing. Install all three wheels, then Kernel → Restart Kernel.

NameError: name 'ECsvPipe' is not defined

Missing import. Each Enterprise class comes from its own module — see Step 6.

Cannot call methods on a stopped SparkContext

Two phases were run in separate cells, each creating a new context. Use trainMatch, or restart the kernel between phases.

ConnectionRefusedError: [Errno 111]

The JVM is gone — usually a consequence of the error above, or a driver OOM. Restart the kernel and re-run from Step 5.

java.lang.OutOfMemoryError: Java heap space

Driver heap exhausted building the query plan. Recreate on a *-standard-8 node with spark.driver.memory=12g and confirm the checkpoint directory is set.

ClassNotFoundException on the first Zingg call

A path in spark.jars is wrong. The cluster builds fine regardless — verify paths against gcloud storage ls.

Last updated

Was this helpful?