Skip to content

Instantly share code, notes, and snippets.

@j-thepac
Created November 11, 2021 04:55
Show Gist options
  • Save j-thepac/2da1cbc2caceb713a0bad1ac091d0f90 to your computer and use it in GitHub Desktop.
Save j-thepac/2da1cbc2caceb713a0bad1ac091d0f90 to your computer and use it in GitHub Desktop.
Convert Spark Dataframe Columns to json
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark=SparkSession
.builder()
.master("local")
.appName("local")
.getOrCreate()
import spark.implicits._
val df= List(("A",1),("B",2)).toDF("c1","c2")
df.select(to_json(struct($"c1",$"c2")).as("new")).show()
//+-----------------------+
//|new|
//+-----------------------+
//| {"c1":"A","c2":1}|
//| {"c1":"B","c2":2}|
//+-----------------------+
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment