> For the complete documentation index, see [llms.txt](https://docs.zingg.ai/zingg/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.zingg.ai/zingg/connect-your-data/pipes-and-data-connections.md).

# Pipes and Data Connections

What a Zingg pipe is and how you configure one to connect your data.

A Zingg pipe is a configuration object that tells Zingg where to find your source data and where to write resolved output. It is how Zingg connects to your datastore, whether that is a\
CSV file, a Delta table, a Snowflake table, a Parquet path on S3, or any other Spark-supported connector.

You configure a pipe by giving it a name, a format, and the connection properties for your specific datastore. The pipe is then passed to your Zingg arguments object `args.setData()`.\
for input, `args.setOutput()` for output.

### What every pipe configuration contains

Every pipe, whether input or output, has three attributes.

<table><thead><tr><th width="152.70703125" valign="top">Attribute</th><th valign="top">What it does</th></tr></thead><tbody><tr><td valign="top"><code>name</code></td><td valign="top">A unique label for this pipe. Used internally by Zingg to identify the data source or destination in logs and output.</td></tr><tr><td valign="top"><code>format</code></td><td valign="top">The Spark connector format string for your datastore: csv, parquet, delta, net.snowflake.spark.snowflake, jdbc, avro, and others.</td></tr><tr><td valign="top"><code>props</code> / <code>options</code></td><td valign="top">The connection properties are passed to <code>spark.read</code> and<br><code>spark.write</code> path, delimiter, header, credentials, and any connector-<br>specific settings.</td></tr></tbody></table>

### Configuring a pipe

You can configure pipes in two ways. Both work in Community and Enterprise. Use whichever fits your workflow.

{% tabs %}
{% tab title="Community" %}
Create pipe objects and attach them to your arguments object.

### **Community**

```python
from zingg.client import *
from zingg.pipes import *
```

#### **Input Pipe**

```python
inputPipe = CsvPipe(
    "testFebrl",
    "examples/febrl/test.csv",
    schema)
args.setData(inputPipe)
```

#### **Output Pipe**

<pre class="language-python"><code class="lang-python">outputPipe = CsvPipe(
<strong>    "resultFebrl",
</strong>    "/tmp/febrlOutput")
args.setOutput(outputPipe)
</code></pre>

{% endtab %}

{% tab title="Enterprise" %}

#### **Enterprise**

```python
from zinggEC.enterprise.common.epipes import *
from zinggEC.enterprise.common.EArguments import *
```

#### **Input Pipe**

```python
inputPipe = ECsvPipe("testFebrl", "examples/febrl/test.csv", schema)
args.setData(inputPipe)
```

#### **Output Pipe**

```python
outputPipe = ECsvPipe("resultFebrl", "/tmp/febrlOutput")
outputPipe.addProperty("header", "true")
args.setOutput(outputPipe)
```

{% endtab %}

{% tab title="JSON config" %}
Define pipes in the data and output sections of your JSON config file.

### **Community**

```json
{
  "data" : [ {
    "name" : "testFebrl",
    "format" : "csv",
    "props" : {
      "location" : "examples/febrl/test.csv",
      "delimiter" : ",",
      "header" : "false"
    }
  } ],
  "output" : [ {
    "name" : "resultFebrl",
    "format" : "csv",
    "props" : {
      "location" : "/tmp/febrlOutput",
      "delimiter" : ",",
      "header" : "true"
    }
  } ]
}
```

### **Enterprise**

```json
{
  "data" : [ {
    "name" : "testFebrl",
    "format" : "csv",
    "props" : {
      "location" : "examples/febrl/test.csv",
      "delimiter" : ",",
      "header" : "false"
    },
    "schema" : "id string, fname string, lname string, stNo string, add1 string, add2 string, city string, state string, areacode string, dob string, ssn string"
  } ],
  "output" : [ {
    "name" : "OUTPUT_FEBRL",
    "format" : "csv",
    "props" : {
      "location" : "/tmp/zinggOutputNew/",
      "delimiter" : ",",
      "header" : true
    }
  } ],
  "outputStats" : {
    "name" : "stats",
    "format" : "csv",
    "props" : {
      "location" : "/tmp/zinggStats_$ZINGG_DYNAMIC_STAT_NAME",
      "delimiter" : ",",
      "header" : true
    }
  }
}
```

{% endtab %}
{% endtabs %}

{% hint style="success" icon="right-long" %}
`CsvPipe` and the standard `Pipe` class are Community APIs. `ECsvPipe`, `UCPipe`, and the other `E*Pipe` classes are Enterprise APIs that add features like Unity Catalog table access, Lakehouse table support, and stats output. The connection properties and basic pipe operations are identical, only the class name and the import path change.

See [Install Zingg → Snowflake](/zingg/running-zingg/install-zingg.md) for the full setup.
{% endhint %}

### What you can connect Zingg to

Zingg connects to any datastore that has a Spark connector, plus Snowflake natively in Enterprise. The pipe format string is all that changes.

<table><thead><tr><th valign="top">Datastore</th><th valign="top">Format string</th><th valign="top">Edition</th></tr></thead><tbody><tr><td valign="top">CSV / TSV</td><td valign="top">csv</td><td valign="top">All</td></tr><tr><td valign="top">Parquet</td><td valign="top">parquet</td><td valign="top">All</td></tr><tr><td valign="top">Avro</td><td valign="top">avro</td><td valign="top">All</td></tr><tr><td valign="top">JSON</td><td valign="top">json</td><td valign="top">All</td></tr><tr><td valign="top">Delta tables (Databricks)</td><td valign="top">delta</td><td valign="top">All</td></tr><tr><td valign="top">Unity Catalog (Databricks)</td><td valign="top">delta</td><td valign="top">All</td></tr><tr><td valign="top">OneLake / Fabric</td><td valign="top"><code>abfss://</code> path + csv or parquet</td><td valign="top">All</td></tr><tr><td valign="top">Snowflake (as data source via Spark)</td><td valign="top"><code>net.snowflake.spark.snowflake</code></td><td valign="top">All</td></tr><tr><td valign="top">Snowflake (native — no Spark cluster)</td><td valign="top">Configured via Snowflake properties file—no pipe class</td><td valign="top">Enterprise</td></tr><tr><td valign="top">AWS S3</td><td valign="top">csv / parquet via <code>s3a://</code></td><td valign="top">All</td></tr><tr><td valign="top">Google Cloud Storage</td><td valign="top">csv / parquet via <code>gs://</code></td><td valign="top">All</td></tr><tr><td valign="top">BigQuery</td><td valign="top"><code>com.google.cloud.spark.bigquery</code></td><td valign="top">All</td></tr><tr><td valign="top">PostgreSQL / MySQL / JDBC</td><td valign="top"><code>jdbc</code></td><td valign="top">All</td></tr><tr><td valign="top">Cassandra</td><td valign="top"><code>org.apache.spark.sql.cassandra</code></td><td valign="top">All</td></tr><tr><td valign="top">MongoDB</td><td valign="top">mongo</td><td valign="top">All</td></tr><tr><td valign="top">Neo4j</td><td valign="top"><code>org.neo4j.spark.DataSource</code></td><td valign="top">All</td></tr><tr><td valign="top">Exasol</td><td valign="top"><code>com.exasol.spark</code></td><td valign="top">All</td></tr><tr><td valign="top">Redshift</td><td valign="top"><code>jdbc</code> (redshift driver)</td><td valign="top">All</td></tr></tbody></table>

{% hint style="success" icon="right-long" %}
**Read more**:

* For the full connection config including required JARs, props, and code examples for each datastore - [Cloud Warehouses](/zingg/connect-your-data/connect-cloud-warehouses.md) | [Cloud Storage](/zingg/connect-your-data/connect-cloud-storage.md) | [File Formats](/zingg/connect-your-data/connect-file-formats.md) | [Relational Databases](/zingg/connect-your-data/connect-relational-databases.md) | [NoSQL Databases](/zingg/connect-your-data/connect-nosql-databases.md) | [Neo4j](/zingg/connect-your-data/connect-graph-databases-neo4j.md)
* To configure your pipes step by step as part of your Zingg setup - [Configure Zingg](/zingg/running-zingg/configure-zingg.md)
* To understand what Zingg adds to your output alongside your input fields (`Z_CLUSTER`, `Z_MINSCORE`, `Zingg ID` and others) - [Interpret Output Scores](/zingg/interpreting-results/interpret-output-scores.md)
  {% endhint %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.zingg.ai/zingg/connect-your-data/pipes-and-data-connections.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.