Create a Spark session optimized to work with Amazon S3.

SparkSessionS3.scala

import org.apache.spark.sql.SparkSession

object SparkSessionS3 {
    //create a spark session with optimizations to work with Amazon S3.
    def getSparkSession: SparkSession = {
        val spark = SparkSession
            .builder
            .appName("my spark application name")
            .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
            .config("spark.hadoop.fs.s3a.access.key", "my access key")
            .config("spark.hadoop.fs.s3a.secret.key", "my secret key")
            .config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
            .config("spark.hadoop.fs.s3a.multiobjectdelete.enable","false")
            .config("spark.hadoop.fs.s3a.fast.upload","true")
            .config("spark.sql.parquet.filterPushdown", "true")
            .config("spark.sql.parquet.mergeSchema", "false")
            .config("spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version", "2")
            .config("spark.speculation", "false")
            .getOrCreate

        // You can use this hadoop configuration as alternative to spark.hadoop configuration
        spark.sparkContext.hadoopConfiguration.set("fs.s3a.multiobjectdelete.enable","false")
        spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key","my access key")
        spark.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key","my secret key")
    }
}

It depends on when you want to provide the S3 connection configs to Spark.
For example, if you want to use S3 as your Spark warehouse location

.config("spark.sql.warehouse", "s3a://bucket_name/")

so here you need to first provide the S3 connection configs while creating the SparkSession.

.config("spark.hadoop.fs.s3a.access.key", "my access key")
.config("spark.hadoop.fs.s3a.secret.key", "my secret key")
.config("spark.hadoop.fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
.config("spark.hadoop.fs.s3a.multiobjectdelete.enable","false")
.config("spark.hadoop.fs.s3a.fast.upload","true")
.config("spark.sql.warehouse", "s3a://bucket_name/")

And if you want to setup the S3 connection later then you can configure SparkContext after you initialize SparkSession like this,

spark.sparkContext.hadoopConfiguration.set("fs.s3a.multiobjectdelete.enable","false")
spark.sparkContext.hadoopConfiguration.set("fs.s3a.access.key","my access key")
spark.sparkContext.hadoopConfiguration.set("fs.s3a.secret.key","my secret key")

Let me know if it helps.