Skip to content

Instantly share code, notes, and snippets.

@przemek-pokrywka
Created September 4, 2016 17:43
Show Gist options
  • Save przemek-pokrywka/d0fdc5bb38d0befd5028135c28bea661 to your computer and use it in GitHub Desktop.
Save przemek-pokrywka/d0fdc5bb38d0befd5028135c28bea661 to your computer and use it in GitHub Desktop.
The example shown at Scalapolis 2016 in Wrocław (with small improvements). HTML scraping of the conference page done as a one-file executable Scala script.
trait valid_both_in_bash_and_in_scala /* 2>/dev/null
# ^^^ can be replaced by anything that's harmless in Bash and valid top-file-def in Scala.
# Kudos to that attendee (sorry, I don't know your name), who noticed that opportunity. Cheers!
# Making sure Coursier is available
cr=~/.coursier
test -e $cr/cr || (mkdir $cr && wget -q -O $cr/cr https://git.io/vgvpD && chmod +x $cr/cr)
dependencies=(
org.jsoup:jsoup:1.7.2
com.lihaoyi:ammonite-repl_2.11.7:0.5.5 # Mandatory for running Scala script
)
# Generate simple build.sbt for editing in IDEs locally. Run with `--build.sbt` (now also on Macs)
test "$1" == "--build.sbt" && \
printf '%s\n' "${dependencies[@]}" | \
sed 's/\(.*\):\(.*\):\(.*\)/libraryDependencies += "\1" % "\2" % "\3"/g' > build.sbt && \
exit
# Small enhancement to the Scalapolis version. Enabling Ammonite to cache compilation output:
just_scala_file=${TMPDIR:-/tmp}/$(basename $0)
(sed -n '/^object script/,$ p' $0; echo "script.run()") > $just_scala_file
CLASSPATH="$($cr/cr fetch -q -p ${dependencies[*]} )" \
java \
ammonite.repl.Main $just_scala_file # hide Bash part from Ammonite
# and make it run the Scala part
exit $?
# */
object script {
def run() {
import org.jsoup._
import collection.JavaConversions._
val doc = Jsoup.connect("http://konf.scalapolis.pl/").get()
val trs = doc.select("#schedule-content > table > tbody > tr")
val lines = trs map { tr =>
val Seq(left, right) = tr.select("td").toSeq
(left.text, right.select("a").text, right.select("p").text)
}
lines foreach println
}
}
@przemek-pokrywka
Copy link
Author

The script, when made executable by chmod +x ./scraper.scala can be run completely stand-alone on any unix host with Java installed.
It contains following improvements to the way of running Scala (*.sc) scripts described on the Ammonite REPL page (http://www.lihaoyi.com/Ammonite/#RunningScripts):

  • it is completely standalone - just make sure Java's installed, make the file executable and run it. Everything else will be downloaded if necessary, you don't need to care.
  • it's IDE-friendly:
    • all code completion, refactoring, usage search, source downloading and other good stuff is at your command. Just run the script with --build.sbt as 1st arg to generate a simple sbt file that all IDEs love. This lets you import the project effortlessly.
    • nothing gets marked as an error. It's all valid Scala code. It doesn't contain any non-standard syntax like import $ivy.group:artifact:version`` that IDE would complain about. The Bash part is enclosed in Scala comment (IntelliJ lets you even edit it as proper Bash with all IDE support using the "inject language" feature).

Big thanks to @lihaoyi for Ammonite REPL and to @alexarchambault for Coursier.
Thanks for all attendees of my talk at Scalapolis 2016 and especially to that one person who has noticed, that the file can be made valid both for Bash and for Scala. Cheers!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment