Skip to content

Instantly share code, notes, and snippets.

@joshyorko
joshyorko / conf.py
Created July 8, 2024 18:46
Spark Conf 3.5 Nessie, Minio, Spark
conf = (
pyspark.SparkConf()
.setAppName("Iceberg Partitioned Data Write")
.set("spark.jars", jdbc_driver_path) # Include the JDBC driver
.set("spark.jars.packages", "org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.5.2,org.projectnessie.nessie-integrations:nessie-spark-extensions-3.5_2.12:0.91.3,software.amazon.awssdk:bundle:2.17.81,org.apache.hadoop:hadoop-aws:3.3.1") # Include Iceberg, Nessie, AWS SDK, and Hadoop AWS packages
.set("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions,org.projectnessie.spark.extensions.NessieSparkSessionExtensions") # Corrected Spark session extensions
.set("spark.sql.catalog.nessie", "org.apache.iceberg.spark.SparkCatalog")
.set("spark.sql.catalog.nessie.uri", NESSIE_URI)
.set("spark.sql.catalog.nessie.ref", "main")
.set("spark.sql.catalog.nessie.authentication.type", "NONE")

Objective This agent automates the process of creating, configuring, testing, and deploying Robocorp robot projects using RCC CLI commands. It guides users through the entire lifecycle of a Robocorp robot, from initial scaffolding to deployment in Control Room.

Instructions

  1. Scaffold a New Robot Project Prompt the user for the desired robot name. Use the rcc create command to scaffold a new robot project: rcc create <robot_name> Confirm the project creation and navigate to the newly created project directory.
  2. Configure Python Dependencies