Stas Batururimi sbatururimi

Spark Tips & Tricks

If values are integers in [0, 255], Parquet will automatically compress to use 1 byte unsigned integers, thus decreasing the size of saved DataFrame by a factor of 8.
Partition DataFrames to have evenly-distributed, ~128MB partition sizes (empirical finding). Always err on the higher side w.r.t. number of partitions.
Pay particular attention to the number of partitions when using flatMap, especially if the following operation will result in high memory usage. The flatMap op usually results in a DataFrame with a [much] larger number of rows, yet the number of partitions will remain the same. Thus, if a subsequent op causes a large expansion of memory usage (i.e. converting a DataFrame of indices to a DataFrame of large Vectors), the memory usage per partition may become too high. In this case, it is beneficial to repartition the output of flatMap to a number of partitions that will safely allow for appropriate partition memory sizes, based upon the

	To mount a partition at startup for all users, we need an entry in the fstab file. What is happening presently is, the HDD is getting mounted for the user who logs in which gives access permissions to only that user. By adding an entry in the fstab, the partition will be mounted by root with access to all users. this r/w access can be controlled later on.

	sudo blkid lists down all partitions available on your system. Note down the UUID of the NTFS partition that you want to mount at boot. In your case, it seems 00148BDE148BD4D6

	now create a folder, for example sudo mkdir /media/ExtHDD01. This is the folder where your external HDD partition will be mounted at. This folder will be owned by root. To give other users permission to r/w into this folder we need to give the proper permissions. so chmod -R 777 /media/ExtHDD01 would be good enough. Now you need to edit your fstab file. to do so, type the following command.

	sudo nano /etc/fstab
	go to the bottom of the file and add the following line there.

	UUID=001

	% bin/nutch parsechecker -Dplugin.includes='protocol-selenium\|parse-tika' \
	-Dselenium.grid.binary=.../geckodriver \
	-Dselenium.enable.headless=true \
	-followRedirects \
	-dumpText https://nutch.apache.org

	# Modern secure (OpenSSH Server 7+) SSHd config by HacKan
	# Refer to the manual for more info: https://www.freebsd.org/cgi/man.cgi?sshd_config(5)

	# Server fingerprint
	# Regenerate with: ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa -b 4096
	HostKey /etc/ssh/ssh_host_rsa_key
	# Regerate with: ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
	HostKey /etc/ssh/ssh_host_ed25519_key

	# Log for audit, even users' key fingerprint

	# terminal color
	export PS1="\[\033[00;35m\][GPU_Andrew] \[\033[00:32m\]\\u:\\W\\$\[\033[00m\] "

	# alternative
	# export PS1="\[\033[00;36m\][\\d, \\t] \[\033[00:32m\]\\u:\\W\\$\[\033[00m\] "

	# vi as vim
	alias vi=vim

	# alias for ssh procy (uncomment if needed)

	[MASTER]

	# Specify a configuration file.
	#rcfile=

	# Python code to execute, usually for sys.path manipulation such as
	# pygtk.require().
	#init-hook=

	# Profiled execution.

	mkdir ~/jupyter
	cd ~/jupyter
	wget https://github.com/graphframes/graphframes/archive/release-0.6.0.zip
	unzip release-0.6.0.zip
	cd graphframes-release-0.6.0
	build/sbt assembly
	cd ..

	# Copy necessary files to root level so we can start pyspark.
	cp graphframes-release-0.6.0/target/scala-2.11/graphframes-assembly-0.6.0-spark2.3.jar .