Scala Script Runner

README.md

Scala Script Runner

Make sure you have coursier installed

chmod +x test.scala
./test.scala

First time it will download scala-compiler (and depending on your config maybe GraalVM) then cache a bootstrap runner. Next time it will run much faster using the cached runner (depending on you config can be native program!).

Understand the shebang

First line of test.scala contains all the dependencies the scala program needs followed by flags passed as is to coursier bootstrap.

• --manifest-jar for JVM-based executable
• --native-image for native executable (use GraalVM native image, make sure you have it installed)
• --native for native executable (use Scala native, make sure you have it installed)

scala-script-launcher.py

#!/usr/bin/env python3
import sys
import hashlib
import os
import subprocess
from datetime import datetime
import itertools

filename = sys.argv[-1]
with open(filename, 'r') as f:
  all_code = f.read()

content_hash = hashlib.sha256(all_code.encode()).hexdigest()
scala_code = all_code[1:]

cache_path = os.path.expanduser(f"~/.ivy2/local/com.abdulradi.scala-scripts-launcher/generated/{content_hash}")
launcher_path = f'{cache_path}/launcher'

if not os.path.exists(launcher_path):
  dependencies = list(itertools.takewhile(lambda x: not x.startswith('-'), sys.argv[1:-1]))
  bootstrap_flags = ' '.join(itertools.dropwhile(lambda x: not x.startswith('-'), sys.argv[1:-1]))
  now = datetime.now().timestamp()
  
  os.makedirs(f'{cache_path}/jars', exist_ok=True)
  os.makedirs(f'{cache_path}/ivys', exist_ok=True)

  # Copy of scala code with shebang commented out as it confuses scalac. TODO: Find a better place for this file
  scalacode_path = f'{cache_path}/jars/generated.scala'
  with open(scalacode_path, "w") as f:
    f.write(f'//{all_code}') 

  jar_path = f'{cache_path}/jars/generated.jar'
  class_path = subprocess.check_output(['coursier', 'fetch', '--classpath'] + dependencies).decode('utf-8')

  os.system(f'coursier launch scalac -- {scalacode_path} -d {jar_path} -classpath {class_path} ')
  scala_version = subprocess.check_output(['coursier', 'launch', 'scalac', '--', '-version']).decode('utf-8')[23:-55] # Drop the copyrights and extract version number, probably very fragile and might break with dotty 
  [scala_version_major, scala_version_minor, scala_version_patch] = scala_version.split('.')
  with open(f'{cache_path}/ivys/ivy.xml', "w") as f:
    f.write('<?xml version="1.0" encoding="UTF-8"?>\n')
    f.write('<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">\n')
    f.write(f'  <info organisation="com.abdulradi.scala-scripts-launcher" module="generated" revision="{content_hash}" status="integration" publication="{now}"></info>\n')
    f.write('  <configurations>\n')
    f.write('    <conf name="compile" visibility="public" description=""/>\n')
    f.write('    <conf name="optional" visibility="public" description=""/>\n')
    f.write('    <conf name="scala-tool" visibility="private" description=""/>\n')
    f.write('  </configurations>\n')
    f.write('  <dependencies>\n')
    f.write(f'    <dependency org="org.scala-lang" name="scala-compiler" rev="{scala_version}" conf="scala-tool->default,optional(default)"/>\n')
    f.write(f'    <dependency org="org.scala-lang" name="scala-library" rev="{scala_version}" conf="scala-tool->default,optional(default);compile->default(compile)"/>\n')
    for dependency in dependencies:
      if '::' in dependency:
        [org, name_rev] = dependency.split('::')
        [name_unversioned, rev] = name_rev.split(':')
        name = f'{name_unversioned}_{scala_version_major}.{scala_version_minor}'
      else:
        [org, name, rev] = dependency.split(':')

      f.write(f'    <dependency org="{org}" name="{name}" rev="{rev}" conf="compile->default(compile)"/>\n')

    f.write('  </dependencies>\n')
    f.write('</ivy-module>\n')

  os.system(f'coursier bootstrap com.abdulradi.scala-scripts-launcher:generated:{content_hash} {bootstrap_flags} -o {launcher_path} --quiet')


os.system(launcher_path)

test.scala

#!/usr/bin/env python3 scala-script-launcher.py io.circe::circe-generic:0.12.3 io.circe::circe-parser:0.12.3 --manifest-jar

import io.circe._, io.circe.generic.auto._, io.circe.parser._, io.circe.syntax._

sealed trait Foo
case class Bar(xs: Vector[String]) extends Foo
case class Qux(i: Int, d: Option[Double]) extends Foo

object Main {
  def main(args: Array[String]): Unit = {
    val foo: Foo = Qux(1, Some(14.0))
    val json = foo.asJson.noSpaces
    println(json)
    val decodedFoo = decode[Foo](json)
    println(decodedFoo)
  }
}

Do I understand correctly that the Scala compiler produces a jar as a result?

I want to use Coursier in a similar way in https://github.com/tsk-tsk/tsk-tsk, as an opt-in, because the native compilation may take long.

Do I understand correctly that the Scala compiler produces a jar as a result?

Yes. Then Coursier takes that jar and creates a bootstrap around it.

I want to use Coursier in a similar way in https://github.com/tsk-tsk/tsk-tsk, as an opt-in, because the native compilation may take long.

Yea I agree. It's opt-in here too, user has to pass --native or --native-image in the shebang.

@tabdulradi, I think that longer-term the hack/solution might be better off not relying on Coursier's native image generation because one may run into troubles in case of dependency exclusions. One would need to construct the Ivy module descriptor (or Maven pom) very carefully in order not to end up with an unwanted version of a dependency conflict.

So in TSK I plan to use Coursier just for fetching GraalVM (and possibly also for running gu to install the native-image tool) but to use the native-image on my own, supplying the classpath resolved by Coursier basing on all the exclusions I care about.

/cc @alexarchambault

@tabdulradi, have you had any success with compiling your example script to a native image? I've followed the steps you gave and unfortunately GraalVM was "Aborting stand-alone image build due to unsupported features" (also I needed to pass -S option to the /usr/bin/env program on Linux).

I added support for GraalVM to TSK recently and I tried to turn your example script into a native binary (by saving, chmodding +x and running the following):

// 2> /dev/null \
/*
dependencies='
  io.circe::circe-generic:0.12.3
  io.circe::circe-parser:0.12.3
'
native=true
source $( curl -L git.io/boot-tsk | sh ); run
*/

import io.circe._, io.circe.generic.auto._, io.circe.parser._, io.circe.syntax._

sealed trait Foo
case class Bar(xs: Vector[String]) extends Foo
case class Qux(i: Int, d: Option[Double]) extends Foo

object Main {
  def main(args: Array[String]): Unit = {
    val foo: Foo = Qux(1, Some(14.0))
    val json = foo.asJson.noSpaces
    println(json)
    val decodedFoo = decode[Foo](json)
    println(decodedFoo)
  }
}

but I encountered the same issues. I guess that one needs to come up with some extra GraalVM-specific descriptors that mention all uses of reflection and class loading for the ahead-of-time compilation to work.

I've recently read https://www.lightbend.com/blog/writing-kubectl-plugins-with-scala-or-java-with-fabric8-kubernetes-client-on-graalvm where Andrea mentions use of GraalVM JVM agent library in order to generate these descriptors.
Due to use of a custom "alternative main" (something that exercises the likely-problematic features of a program) this method may be not appropriate for cs bootstrap and/or for scripts, but I'm curious of your thoughts @tabdulradi, @alexarchambault