After much todo, trying to get powershell to run julia like it would in bash, I got this working:
& julia -i -e ('println(\"Hello World\")')
& julia -i -e ('include(\"path-to-script.jl\"); function_in_script(); exit()')
& julia -i -e "path-to-script.jl"
Note the parenthesis and the single quotes and still escaping the double quotes.
Spaces, quotes and more are weird special characters to the arg parsing of powershell.
If you don't include the -i argument, then you can't exit the script with Ctrl-C.