AWS S3
Zingg can use AWS S3 as a source and sink
Steps to run zingg on S3
Set a bucket, for example - zingg28032023 and a folder inside it,for example - zingg
Create aws access key and export via env vars (ensure that the user with below keys has read/write access to above)
export AWS_ACCESS_KEY_ID=<access key id>
export AWS_SECRET_ACCESS_KEY=<access key>
(if mfa is enabled AWS_SESSION_TOKEN env var would also be needed )Download hadoop-aws-3.1.0.jar and aws-java-sdk-bundle-1.11.271.jar via maven
Set above in zingg.conf spark.jars=//hadoop-aws-3.1.0.jar,//aws-java-sdk-bundle-1.11.271.jar
Run using below commands
./scripts/zingg.sh --phase findTrainingData --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase label --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase train --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
./scripts/zingg.sh --phase match --properties-file config/zingg.conf --conf examples/febrl/config.json --zinggDir s3a://zingg28032023/zingg
Model location
Models etc. would get saved in
Amazon S3 > Buckets > zingg28032023 > zingg > 100
Last updated
Was this helpful?