YannMoisan

Mon incroyable 9.3.

Agenda

• 13-14/06 Festival Et 20 l'été
• Ven 19/06 : 19h TÂMÁN - Moh ! Kouyaté & Mama Djessou Diabaté Noisy-le-Sec
• 20-21/06 : Week-end de ouf Montreuil
• Jeu 25/06 : 19h Mauvais joueurs - Académie Frattelini / Mehdi Kerkouche Noisy-le-Sec
• 30/06-05/07 : Biennale Urbaine de Spectacles Pantin
• Mer 01/07 : 19h30 Surcouf - Compagnie Sacékripa Noisy-le-Sec

YannMoisan

I was wondering if catalyst optimize consecutive unionAll (so that the number of shuffles will be limited).

So let's write a test

@ val dfs = (1 to 5).map(i => sc.parallelize((1 to 3).map(j => i + 3*j)).toDF("a"))
dfs: collection.immutable.IndexedSeq[org.apache.spark.sql.DataFrame] = Vector([a: int], [a: int], [a: int], [a: int], [a: int])
@ val merged = dfs.reduceLeft(_ unionAll _)
merged: org.apache.spark.sql.DataFrame = [a: int]

And show the plan 

YannMoisan

Context : Encryption of an external drive that need to be accessed from OSX and Linux

The drive has an existing NTFS partition

I use veracrypt. It's open source, and seems to be the de facto standard nowadays.

On OSX, I installed OSX Fuse.

I create a volume within the partition, and the problems begin with the choice of the filesystem.
Firstly, this choice depends on the OS from which I create the volume :