Story: Writing Scripts with Go

go-shebang-story.md

Story: Writing Scripts with Go

This is a story about how I tried to use Go for scripting. In this story, I’ll discuss the need for a Go script, how we would expect it to behave and the possible implementations; During the discussion I’ll deep dive to scripts, shells, and shebangs. Finally, we’ll discuss solutions that will make Go scripts work.

Why Go is good for scripting?

While python and bash are popular scripting languages, C, C++ and Java are not used for scripts at all, and some languages are somewhere in between.

Go is very good for a lot of purposes, from writing web servers, to process management, and some say even systems. In the following article, I argue, that in addition to all these, Go can be used, easily, to write scripts.

What makes Go good for scripts?

• Go is simple,readable, and not too verbose. This makes the scripts easy to maintain, and relatively short.
• Go has many libraries, for all sorts of uses. This makes the script short and robust, assuming the libraries are stable and tested.
• If most of my code is written in Go, I prefer to use Go for my scripts as well. When a lot of people are collaborating code, it is easier if they all have full control over the languages, even for the scripts.

Go is 99% There Already

As a matter of fact, you can already write scripts in Go. Using Go’s run subcommand: if you have a script named my-script.go, you can simply run it with go run my-script.go.

I think that the go run command, needs a bit more attention in this stage. Let’s elaborate about it a bit more.

What makes Go different from bash or python is that bash and python are interpreters - they execute the script while they read it. On the other hand, when you type go run, Go compiles the Go program, and then runs it. The fact that the Go compile time is so short, makes it look like it was interpreted. it is worth mentioning “they” say “go run is just a toy", but if you want scripts, and you love Go, this toy is what you want.

So we are good, right?

We can write the script, and run it with the go run command! What’s the problem? The problem is that I'm lazy, and when I run my script I want to type ./my-script.go and not go run my-script.go.

Let’s discuss a simple script that has two interactions with the shell: it gets an input from the command line, and sets the exit code. Those are not all the possible interactions (you also have environment variables, signals, stdin, stdout and stderr), but two problematic ones with shell scripts.

The script writes “Hello”, and the first argument in the command line, and exits with the code 42:

package main

import (
    "fmt"
    "os"
)

func main() {
    fmt.Println("Hello", os.Args[1])
    os.Exit(42)
}

The go run command behaves a bit weird:

$ go run example.go world
Hello world
exit status 42
$ echo $?
1

We’ll discuss that later on.

The go build can be used. This is how you would run it using the go build command:

$ go build
$ ./example world
Hello world
$ echo $?
42

Current workflow with this script looks like this:

$ vim ./example.go
$ go build
$ ./example.go world
Hi world
$ vim ./example.go
$ go build
$ ./example.go world
Bye world

What I want to achieve, is to run the script like this:

$ chmod +x example.go
$ ./example.go world
Hello world
$ echo $?
42

And the workflow I would like to have is this:

$ vim ./example.go
$ ./example.go world
Hi world
$ vim ./example.go
$ ./example.go world
Bye world

Sounds easy, right?

The Shebang

Unix-like systems support the Shebang line. A shebang is a line that tells the shell what interpreter to use to run the script. You set the shebang line according to the language that you wrote your script in.

It is also common to use the env command as the script runner, and then an absolute path to the interpreter command is not necessary. For example: #! /usr/bin/env python to run the python interpreter with the script. For example: if a script named example.py has the above shebang line, and it is executable (you executed chmod +x example.py), then by running it in the shell with the command ./example.py arg1 arg2, the shell will see the shebang line, and starts this chain reaction:

The shell runs /usr/bin/env python example.py arg1 arg2. This is actually the shebang line plus the script name plus the extra arguments. The command invokes /usr/bin/env with the arguments: /usr/bin/env python example.py arg1 arg2. The env command invokes python with python example.py arg1 arg2 arguments python runs the example.py script with example.py arg1 arg2 arguments.

Let’s start by trying to add a shebang to our go script.

1. First Naive Attempt:

Let's start with a naive shebang that tries to run go run on that script. After adding the shebang line, our script will look like this:

#! /usr/bin/env go run
package main

import (
    "fmt"
    "os"
)

func main() {
    fmt.Println("Hello", os.Args[1])
    os.Exit(42)
}

Trying to run it results in:

Output:

$ ./example.go
/usr/bin/env: ‘go run’: No such file or directory

What happened?

The shebang mechanism sends "go run" as one argument to the env command as one argument, and there is no such command, typing which “go run” will result in a similar error.

2. Second Attempt:

A possible solution could be to put #! /usr/local/go/bin/go run as the shebang line. Before we try it out, you can already spot a problem: the go binary is not located in this location in all environments, so our script will be less compatible with different go installations. Another solution is to use alias gorun="go run", and then change the shebang to #! /usr/bin/env gorun, in this case we will need to put the alias in every system that we run this script.

Output:

$ ./example.go
package main:
example.go:1:1: illegal character U+0023 '#'

Explanation:

OK, I have good news and bad news, what do you want to hear first? We’ll start with the good news :-)

• The good news are that it worked, go run command was invoked with our script
• The bad news: the hash sign. In a lot of languages the shebang line is ignored as it starts with a comment line indicator. Go compiler fails to read the file, since the line starts with an "illegal character"

3. The Workaround:

When no shebang line is present, different shells will fallback to different interpreters. Bash will fallback to run the script with itself, zsh for example, will fallback to sh. This leaves us with a workaround, as also mentioned in StackOverflow.

Since // is a comment in Go, and since we can run /usr/bin/env with //usr/bin/env (// == / in a path string), we could set the first line to:

//usr/bin/env go run "$0" "$@"

Result:

$ ./example.go world
Hi world
exit status 42
./test.go: line 2: package: command not found
./test.go: line 4: syntax error near unexpected token `newline'
./test.go: line 4: `import ('
$ echo $?
2

Explanation:

We are getting close: we see the output but we have some errors and the status code is not correct. Let's see what happened here. As we said, bash did not meet any shebang, and chose to run the script as bash ./example.go world (this will result in the same output if you'll try it). That's interesting - running a go file with bash :-) Next, bash reads the first line of the script, and ran the command: /usr/bin/env go run ./example.go world. "$0" Stands for the first argument and is always the name of the file that we ran. "$@" stands for all the command line arguments. In this case they were translated to world, to make: ./example.go world. That's great: the script ran with the right command line arguments, and gave the right output.

We also see a weird line that reads: "exit status 42". What is this? If we would try the command ourselves we will understand:

$ go run ./example.go world
Hello world
exit status 42
$ echo $?
1

It is stderr written by the go run command. Go run masks the exit code of the script and returns code 1. For further discussion about this behavior read here Github issue.

OK, so what are the other lines? This is bash trying to understand go, and it isn’t doing very well.

4. Workaround Improvement:

This StackOverflow page suggests to add `;exit "$?" to the shebang line. this will tell the bash interpreter not to continue to the following lines.

Using the shebang line:

//usr/bin/env go run "$0" "$@"; exit "$?"

Result:

$ ./test.go world
Hi world
exit status 42
$ echo $?
1

Almost there: what happened here is that bash ran the script using the go run command, and immediately after, exited with the go run exit code.

Further bash scripting in the shebang line, for sure can remove the stderr "exit status" message, even parse it, and return it as the program exit code.

However:

• Further bash scripting means longer, and exhausting shebang line, which is supposed to look as simple as #! /usr/bin/env go.
• Lets remember that this is a hack, and I don't like that this is a hack. After all, we wanted to use the shebang mechanism - Why? Because it's simple, standard and elegant!
• That’s more or less the point where I stop using bash, and start using more comfortable languages as my scripting languages (such as Go :-) ).

Lucky Us, We Have gorun

gorun does exactly what we wanted. You put it in the shebang line as #! /usr/bin/env gorun, and make the script executable. That’s it, You can run it from your shell, just as we wanted!

$ ./example.go world
Hello world
$ echo $?
42

Sweet!

The Caveat: Compilability

Go fails compilation when it meets the shebang line (as we saw before).

$ go run example.go
package main:
example.go:1:1: illegal character U+0023 '#'

Those two options can’t live together. We must choose:

• Put the shebang and run the script with ./example.go.
• Or, remove the shebang and run the script with go run ./example.go.

You can’t have both!

Another issue, is that when the script lies in a go package that you compile. The compiler will meet this go file, even though it is not part of the files that are needed to be loaded by the program, and will fail the compilation. A workaround for that problem is to remove the .go suffix, but then you can’t enjoy tools such as go fmt.

Final Thoughts

We’ve seen the importance of enabling writing scripts in Go, and we’ve found different ways to run them. Here is a summary of the findings:

Type	Exit Code	Executable	Compilable	Standard
go run	✘	✘	✔	✔
gorun	✔	✔	✘	✘
// Workaround	✘	✔	✔	✔

Explanation: Type: how we chose to run the script. Exit code: after running the script it will exit with the script’s exit code. Executable: the script can be chmod +x. Compilable: the script passes go build Standard: the script doesn’t need anything beside the standard library.

As it seems, there is no perfect solution, and I don’t see why we shouldn’t have one. It seems like the easiest, and least problematic way to run Go scripts is by using the go run command. It is still too ‘verbose’ to my opinion, and can’t be “executable”, and the exit code is incorrect, which makes it hard to tell if the script was completed successfully.

This is why I think there is still work do be done in this area of the language. I don’t see any harm in changing the language to ignore the shebang line. This will solve the execution issue, but a change like this probably won't be accepted by the Go community.

My colleague brought to my attention the fact that the shebang line is also illegal in javascript. But, in node JS, they added a strip shebang function which enables running node scripts from the shell.

It would be even nicer, if gorun could come as part of the standard tooling, such as gofmt and godoc.

Thanks for reading,

If you find this interesting, please, ✎ comment below or ★ star above ☺

For more stuff: gist.github.com/posener.

Cheers, Eyal

One solution is to create an "interpreter" that makes a copy of the file without the shebang line, and then go runs that file.

go-script:

#!/bin/sh

tmpfile=$(mktemp --suffix=.go /tmp/go-script.XXXXXX)
tail --lines=+2 "$@" >$tmpfile
go run $tmpfile
rm $tmpfile

Then one can write

#!/path/to/go-script
package main

func main() {
        println("It works!")
}

Some possible improvements of go-script:

• only remove first line if it really begins with #!
• make sure the temporary file is deleted even if program fails
• allow more scripty code by adding package main and wrapping non-declarations in func main() { ... }

Ok why not use Go build or go install. If the GOPATH is setup correctly you would not need to add a SHEBANG. Go allows you to compile to any platform. Your Go workspace would have /bin /pkg /src

Build your Go app/script with Go Build /src/srcdirectory and it will place an executable file in your /bin directory. You can then take the same Code and then Build it with the command GOOS=windows in front of your build command it will create an exe for windows.

Once you compile your code you can easily run the command without using the run command in GO.

Please accept my apologies If I am misunderstanding something here ..

Well written, I like it. 👍

Suggesting that the compiles consider as comment a shebang line as first line of a package main file would avoid the need to copy the file to strip off the shebang line. The problem of scripts are the access to packages. The script needs to be in the GOPATH dir if we want to be able to use packages. We can't have go scripts in a random directory.

I have been using go for writing scripts for some time.

To make this POSIX compliant @thommey and me figured out, that the following would work and check all the boxes.

// 2>/dev/null;/usr/bin/go run $0 $@; exit $?

Contrary to the assumption in the beginning, // is not necessarily equal to / in all spots of a path. POSIX allows the root of // to be interpreted in an implementation-defined manner (cygwin uses this for example). Three or more slashes would be considered equal to / again, though.

Thank you I liked this a lot!! 💃

https://blog.cloudflare.com/using-go-as-a-scripting-language-in-linux/

which is an extended and amended answer, that @vhodges provided

hello.go

package main

import `fmt`

func main() {
    fmt.Println(`hello, world!`)
}

copy these contents to a file named after the go entry point.. without the .go extension
hello

#!/usr/bin/env bash

cat > $BASH_SOURCE.makefile << EOF
$BASH_SOURCE.goc : $BASH_SOURCE.go
	go build -o $BASH_SOURCE.goc $BASH_SOURCE.go

EOF

make -s -f $BASH_SOURCE.makefile
unlink $BASH_SOURCE.makefile
$BASH_SOURCE.goc

$ chmod 755 ./hello
$./hello
hello there, world!

Typo:

Your Current workflow with this script looks like this: section incorrectly has the commands with a .go suffix.

This article has an explaination of using the binfmt stuff in Linux to do what you want: Using Go as a scripting language in Linux.

I see some problems, though:

• How do you ensure the Go build environment is on all the boxes you want to run scripts on?

  It's over 300MiB which is big for some use ecases.

• How do you ensure your libraries work and are the right version?

  gopkg.in helps somewhat, but not all libraries use this system for version management.

  vgo will eventually help a bunch with this.

• Is it acceptable that your go scripts fail when the internet or github.com are down?

  bash scripts will run whether github.com is up or down. Python and Ruby scripts as well, assuming you pulled all the libraries down before hand.

• How do you handle Go versions?

  As go evolves, you could end up using a feature that only works in older or newer versions of Go.

  This would cause more runtime versioning problems, such as we have with Ruby and Python scripts. There was a bunch of problems when macOs switched from Ruby 1.9.x to 2.x.

Anyway, I think it is a neat idea but I think the extra step of compiling and pushing it out to a directory in the PATH makes more sense at this time. 😃

You can also do this:

$ go build hello.go && ./hello world
Hello world

This gives the right status code and it's a "standard" solution:

$ go build hello.go && ./hello world
Hello world
$ echo $?
42

It's a bit more to type out, but you really only have to do it every once in a while. It also 1) doesn't print out the exit status 42 line, and 2) doesn't require a "shebang" line in your file.

It feels like this is the ideal solution for development. Once you're done writing/modifying your script you can add $GOPATH/bin to your PATH and just go install it:

$ go install
$ cd ~
$ hello world; echo $?
Hello world
42

Another option is to auto-deduce the location of the go command, as follows:

//$(which go) run $0 $@; exit $?
package main

import `fmt`

func main() {
    fmt.Println(`hello, world!`)
}

Assuming, for example, go is on /usr/local/bin/go, this runs ///usr/local/bin/go (with three leading '/'). This works well on Linux & BSD. See also earlier comment by @Grauwolf.

@hanbang-wang

Compile language and script language seem to be contradictory at the first place... But nice work

nodejs and web browsers compile Javascript (a scripting language) into bytecode before its executed.

// 2>/dev/null; e=$(mktemp); go build -o $e "$0"; $e "$@" ; r=$?; rm $e; exit $r

Ugly, but checks all boxes; leaves no executables in tmp; can be used concurrently; provides return code.

Interesting, just thinking how to do same thing in Go like we always done in bash script, like grep, sed, awk commands etc.

Mistake in the second attempt section: alias gorun="go run" will not make #!/usr/bin/env gorun work. Aliases are only in the shell, while /usr/bin/env is an external program, and does not see shell aliases. It would only work if you have a separate script installed in your PATH called gorun.

@shlomi-noach Don't need which in the shell - the shell already does PATH lookup when executing go. Unless you're trying to bypass functions/aliases which are defined for non-interactive and non-login shells - but any functions or aliases that you would want to bypass ought to be defined in a way that only login/interactive shells pick them up, so if that's a problem, it's a system configuration bug, not a bug in your script.

Also, the // "hashbang" and doesn't actually work portably, it only "works" in shells like bash which just assume the file is a shell script and execute it themselves when executing the file "for real" through the operating system fails. Also as some people said, POSIX permits and some systems actually use the feature of // at the beginning of a path meaning something special (but /// or more slashes are equal to / everywhere, and POSIX requires it).

These portability nitpicks might be fine, but worth bearing in mind and making explicit, both because for some people it just won't work, and because not making that clear that it only works if called from shells that do that (instead of when called from any program like a real executable) can mislead people about how things work, and then the misunderstanding spreads, and then people make more code that breaks in ways people don't understand in situations they don't expect, etc.

Also "hashbang"/"shebang" means precisely #! - please don't call // a hashbang/shebang - that's misusing the term in a way that further contributes to the misunderstanding I am talking about.

My 2 cents.

/// 2>/dev/null; go run $0 $@; exit $?

Use triple /'s as suggested, while using the built in PATH to locate the go binary. exit $? will however not show the exit signal from the script though.

So here is my attempt at getting the correct exit signal as well:

/// 2>/dev/null; t=$(mktemp) && go run $0 $@ 2> "$t"; g=$?; e=$(tail -n1 "$t" | grep -E '^exit status [0-9]+$' | cut -d' ' -f3); cat "$t" && rm "$t"; test -n "$e" && exit $e || exit $g

Should also be POSIX compliant. Feel free to improve it.

Thank you all for the useful solution!
I had needed to made system calls from the "Bash-Script", so here an script example I will share with you.

@posener figured it will be of interest to you that go run has a corner case on how it proxies STDIN/OUT/ERR to the underlying "program". TLDR: it will not propagate closures. More details here: golang/go#39172 Sigh...

@dxlr8r commented on May 19, 2020, 10:15 PM GMT+4:30:

My 2 cents.

/// 2>/dev/null; go run $0 $@; exit $?

Use triple /'s as suggested, while using the built in PATH to locate the go binary. exit $? will however not show the exit signal from the script though.

So here is my attempt at getting the correct exit signal as well:

/// 2>/dev/null; t=$(mktemp) && go run $0 $@ 2> "$t"; g=$?; e=$(tail -n1 "$t" | grep -E '^exit status [0-9]+$' | cut -d' ' -f3); cat "$t" && rm "$t"; test -n "$e" && exit $e || exit $g

Should also be POSIX compliant. Feel free to improve it.

I used /// 2>/dev/null; gorun $0 $@; exit $?, and the exit code was correct, without any additional work.

If you're satisfied with the last example, then you should do this instead:

/// 2>/dev/null; exec gorun "$0" "$@"

Because

1. Always quote your shell variables! Unless you know of a reason why you have to leave them unquoted. If you don't follow this rule your scripts will break eventually.
2. You don't need the exit if you just want to exit with the status of the last command since the shell does that automatically.
3. Call the last command you want to execute with exec so that the shell process becomes that process instead of waiting around for it as an intermediate parent process.

How do I structure the folder? I mean hello1.go and hello2.go, both with func main() cannot be in the same folder...

How do I structure the folder? I mean hello1.go and hello2.go, both with func main() cannot be in the same folder...

You can provide the go run command only the files you want. So in your case: go run hello1.go

How do I structure the folder? I mean hello1.go and hello2.go, both with func main() cannot be in the same folder...

You can provide the go run command only the files you want. So in your case: go run hello1.go

Although it does run, VSCode throws warning.

As far as I understand, when using the same package name, everything in the same folder must not have the same name.

Also the case with Kotlin, and perhaps Java (which has jbang).

A point being lost in many of these suggested solutions is that ordinary shell and scripting-language scripts do not require the user to have write access to any directory except possibly a tmpdir; on embedded systems where the OS partition(s) are read-only, /tmp is sometimes the only directory the user has write access to (and /tmp is probably a tmpfs (or other ephemeral RAM disk) which may get purged every time a program exits).

Even scripting languages that write cached precompilations by default (like Python) generally simply silently ignore a failure to write cached bytecode. (Doing a quick perusal, it looks like some interpreters will fail at compile-time if out of date cache files are present & the interpreter can’t replace them with an update, while others will simply silently stop using such non-overwritable out-of-date cached files.)

In the case of Go, I don’t think you can get around having either a writable home directory or other writable $GOPATH. So a more truly portable solution would need to deal with this, presumably by setting GOPATH to a directory in a writable tmpdir.

There is a Unix & Linux Stack Exchange question about exactly this problem—unless something has changed (or no one with the knowledge answered), if the running user has no file creation/writing rights at all, the Go build system itself would need to be changed to allow use of go run (or gorun etc.).

(But to add one final wrinkle: on some OS’s by default, and on most modern OS’s with fine-grained capability controls, for security reasons you may not be able to run arbitrary executable blobs in memory that haven’t been written to “disk”—or whatever filesystem storage you have—at all! So Go would need the capability to act like a modern bytecode-compiling interpreter, rather than a compiler—simply being fast enough to seem like an interpreter—the thing that motivates and enables this whole use case—wouldn’t be enough! This may be a niche restriction right now, but it’s already hard-and-fast on the OS’s of most smartphones in the world.)

Run shell scripts having concatenated binary content.¹

POSIX was recently revised to require this behavior:

"The input file may be of any type, but the initial portion of the
file intended to be parsed according to the shell grammar (XREF to
XSH 2.10.2 Shell Grammar Rules) shall consist of characters and
shall not contain the NUL character. The shell shall not enforce
any line length limits."

{ ... }

"Earlier versions of this standard required that input files to the
shell be text files except that line lengths were unlimited.
However, that was overly restrictive in relation to the fact that
shells can parse a script without a trailing newline, and in
relation to a common practice of concatenating a shell script
ending with an 'exit' or 'exec $command' with a binary data payload
to form a single-file self-extracting archive²."

POSIX me harder

TLDR: POSIX says we shall allow execution of scripts with concatenated binary and suggests checking a line exists before the first NUL character with a lowercase letter or expansion.

This would for example, allow for go to be used as a scripting language for shell scripts, properly.³

References

Initial source: jart/zsh@94a4bc1

Footnotes

1. https://justine.lol/cosmopolitan/index.html ↩

2. http://austingroupbugs.net/view.php?id=1250 ↩

3. http://austingroupbugs.net/view.php?id=1226#c4394 ↩

Why not fork go and allow normal shebang lines in its files?