Pingkai Liu kindofblue

kindofblue / explain_so

Created December 27, 2023 21:50 — forked from jzhuge/explain_so

Writing to iceberg table with sort order after coalesce

	== Parsed Logical Plan ==
	OverwriteByExpression RelationV2[id#61L] prodhive.jzhuge.wso_244_so_1633558130, true, Map(matchByName -> true), true
	+- Repartition 4, false
	+- Sort [id#3L ASC NULLS FIRST], true
	+- Project [id#0L AS id#3L]
	+- Range (0, 100000, step=1, splits=Some(6))

	== Analyzed Logical Plan ==
	OverwriteByExpression RelationV2[id#61L] prodhive.jzhuge.wso_244_so_1633558130, true, Map(matchByName -> true), true
	+- Repartition 4, false

kindofblue / merge-schemas.scala

Created December 18, 2023 19:12 — forked from eduardorost/merge-schemas.scala

Merge Schema with structs

	import org.apache.spark.SparkConf
	import org.apache.spark.sql.SparkSession
	import org.apache.spark.sql.types._
	import org.slf4j.{Logger, LoggerFactory}

	object Main {
	val logger: Logger = LoggerFactory.getLogger(this.getClass)

	private lazy val sparkConf: SparkConf = new SparkConf()
	.setMaster("local[*]")

kindofblue / idea.vmoptions

Created May 25, 2022 18:03

Better performance vmoptions for 2020.1 intellij

	-Xms1024m
	-Xmx3072m
	-Xss64m
	-Dfile.encoding=UTF-8
	-Djava.net.preferIPv4Stack=true
	-Dsun.io.useCanonCaches=false
	-XX:+AggressiveOpts
	-XX:+AlwaysPreTouch
	-XX:+CMSClassUnloadingEnabled
	-XX:+CMSIncrementalMode

kindofblue / CustomZenPackInstall__init__.py

Created March 29, 2020 20:23 — forked from James-Newman/CustomZenPackInstall__init__.py

	## This is an example of executing code when your zenpack gets installed (or removed)
	## This __init__.py file should be located at ZenPacks.namespace.Name/ZenPacks/namespace/Name/__init__.py
	## This code should be carefully added to your zenpack so as not to accidently overwrite key methods (



	'''
	Here begins the non default code
	'''
	# Import all the things.