You may have thought of running nightmare on AWS Lambda. But before we can run it on Lambda, we need first to make it run on Amazon Linux.
According to AWS Documentation on Lambda Execution Environment and available Libraries we would need this AMI image with this alias amzn-ami-hvm-2016.03.3.x86_64-gp2. Keep in mind that AMI-image-id for this instance would be different in different regions (eg):
- In
eu-west-1-ami-f9dd458a - In
us-east-1-ami-6869aa05
But you can find the right one in your "Community AMIs" section of "EC2 Launch Instance wizard".
If in your region you find more than one image with this name, you need to pick one with with description "Amazon Linux AMI 2016.03.3 x86_64 HVM GP2".
So now you can launch the Amazon Linux AMI in your preferred way, I would launch Amazon Linux Image via AWS CLI
Now let's connect to the instance via
ssh -i ~/.ssh/my-amazon-linux-keypair.pem [email protected]
__| __|_ )
_| ( / Amazon Linux AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
22 package(s) needed for security, out of 80 available
Run "sudo yum update" to apply all updates.
Amazon Linux version 2016.09 is available.
$
Now let's prepare instance and install some basic tools we need to try to run nightmare. (Note that we will use node 4.3.2 because that's what Lambda Environment docs say.
sudo yum update -y
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" # This loads nvm
nvm install v4.3.2
node -v
npm -v
Now let's install nightmare and try running sample
mkdir nightmare-test && cd nightmare-test
npm install nightmare # This will automatically install electron executable as dependency
So let's try running example.js:
[~/nightmare-test]$ node node_modules/nightmare/example.js
# Nightmare will return without any output on the console. Let's add debug flag:
[~/nightmare-test]$ DEBUG=nightmare node node_modules/nightmare/example.js
nightmare queuing process start +0ms
nightmare queueing action "goto" for http://yahoo.com +3ms
nightmare queueing action "type" +2ms
nightmare queueing action "click" +0ms
nightmare queueing action "wait" +0ms
nightmare queueing action "evaluate" +0ms
nightmare running +1ms
nightmare electron child process exited with code 127: command not found - you may not have electron installed correctly +19ms
nightmare electron child process not started yet, skipping kill. +1ms
Now we get output like below, from which we can conclude that there's problem with running electron executable. But still we don't know what is the root cause of the problem. So let's try to run the electron executable manually:
[ec2-user@ip-172-31-5-1 nightmare-test]$ ./node_modules/nightmare/node_modules/electron/dist/electron
electron: error while loading shared libraries: libgtk-x11-2.0.so.0: cannot open shared object file: No such file or directory
Bingo! The dependency error was the first informative error and told us that electron can't run because of missing static library libgtk (which is installed by default on most of Desktop Ubuntu distros, but isn't always available on server distros, on CentOS or on Amazon Linux or Lambda).
It is logical to assume that in case one library is missing, there will be more. Let's check
$ cd node_modules/nigthmare/node_modules/electron/dist
[ec2-user@ip-172-31-5-1 dist]$ ldd electron | grep 'not found'
libgtk-x11-2.0.so.0 => not found
libgdk-x11-2.0.so.0 => not found
libatk-1.0.so.0 => not found
libpangocairo-1.0.so.0 => not found
libgdk_pixbuf-2.0.so.0 => not found
libcairo.so.2 => not found
libpango-1.0.so.0 => not found
libXcursor.so.1 => not found
libXdamage.so.1 => not found
libXrandr.so.2 => not found
libXfixes.so.3 => not found
libXss.so.1 => not found
libgconf-2.so.4 => not found
libcups.so.2 => not found
We can see that there's total of 14 static libraries missing. About a year ago Yuanyi wrote great article on How to run electron on Amazon Linux. By now it is a bit outdated and only helps tackle 11 dependencies and misses few important in 2017 parts, but has definitely has saved me a day or two and showed the right approach (and right versions of the libraries to build from source).
However instead of simply rewriting a newer version of the article I have decided to put all of the commands and improvements into a script, so that the process can be automated. The tool is called eltool.sh and is available as gist
So as the next step we can download the tool into the home directory:
curl -o- https://gist.githubusercontent.com/dimkir/52054dfca586cadbd0ecd3ccf55f8b98/raw/2b5ebdf28f6a1aad760b5ab9cc581e8ad12a49f5/eltool.sh > ~/eltool.sh && chmod +x ~/eltool.sh
Now we can proceed with installing missing electron dependencies and compiling from source certain libraries. The syntax of the tool is ./eltool.sh task1 task2 task3 and task names are made to be self explanatory:
$ ./eltool.sh dev-tools # installs gcc compiler and some libs
$ ./eltool.sh dist-deps # we install prebuilt dependencies from Amazon Linux repos by using yum
$ ./eltool.sh centos-deps # we install some prebuil dependencies we can take from CentOS6 repo
# There's still a number of libraries which need to compile from source
$ ./eltool.sh gconf-compile gconf-install
$ ./eltool.sh pixbuf-compile pixbuf-install
$ ./eltool.sh gtk-compile # this will take 3 minutes on t2.small instance
$ ./eltool.sh gtk-install
Now you have all dependencies available in the system directory /usr/local/lib, but some libraries need to be placed(hardlinked) into same directory as electron executable:
$ cd ~/nightmare-test/node_modules/nightmare/node_modules/electron/dist
# Let's create hardlinks to the required libraries
[dist]$ ln -PL /usr/local/lib/libgconf-2.so.4
[dist]$ ln -PL /usr/local/lib/libgtk-x11-2.0.so.0
[dist]$ ln -PL /usr/local/lib/libgdk-x11-2.0.so.0
[dist]$ ln -PL /usr/local/lib/libgdk_pixbuf-2.0.so.0
# or alternatively you can use shorthand
[dist]$ ~/eltool.sh link
At this point in time you should not have any unresolved dependencies for electron, but let's double check:
[dist]$ ldd electron | grep 'not found'
# Output should be empty
Now all missing electron dependencies are present, let's try to run our example.js again:
[nightmare-test]$ node example.js
# Nothing happens, let's try to add DEBUG flag
[nightmare-test]$ DEBUG=nightmare node example.js
nightmare queuing process start +0ms
nightmare queueing action "goto" for http://yahoo.com +3ms
nightmare queueing action "type" +2ms
nightmare queueing action "click" +0ms
nightmare queueing action "wait" +0ms
nightmare queueing action "evaluate" +1ms
nightmare queueing action "screenshot" +0ms
nightmare running +0ms
nightmare electron child process exited with code 1: general error - you may need xvfb +42ms
nightmare electron child process not started yet, skipping kill. +1ms
If you look carefully at this output, and compare it with the debug output we saw in the beginning of the article, you will notice two subtle differences :
- exit code is
1(and not127as was when we had problems with dependencies) - nightmare is suggesting actual error cause - missing Xvfb
So let's follow suggestion and install X-server and Xvfb to the instance:
sudo yum -y install xorg-x11-server-Xorg xterm # x-server
sudo yum -y install xorg-x11-drv-vesa xorg-x11-drv-evdev xorg-x11-drv-evdev-devel # x-drivers
sudo yum -y install Xvfb
# or alternatively you can use shortcut
$ ~/eltool.sh xvfb-install
And now let's finally run nightmare with Xvfb server running in the background:
# Upon successful execution the output should be an url related to nightmare. Any url you see would be sign of success.
[nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js
https://github.com/segmentio/nightmare
# Hurray! This seems to work!
# If you're still not convinced we can run it with DEBUG flag
[nigthmare-test]$ DEBUG=nightmare xvfb-run -a --server-args="-screen 0 1366x768x24" node node_modules/nightmare/example.js
nightmare queuing process start +0ms
nightmare queueing action "goto" for http://yahoo.com +3ms
nightmare queueing action "type" +2ms
nightmare queueing action "click" +0ms
nightmare queueing action "wait" +0ms
nightmare queueing action "evaluate" +1ms
nightmare running +0ms
nightmare electron child process exited with code 0: success! +8s
https://github.com/segmentio/nightmare
CONGRATS! IT FINALLY WORKS! CONGRATS! IT FINALLY WORKS! CONGRATS! IT FINALLY WORKS!
To ensure everything really works, you probably do not want to limit yourself with single line of text returned by the example.js. Let's add screenshot functionality to example.js:
// ~/nightmare-test/example-screenshot.js
var Nightmare = require('nightmare');
var nightmare = Nightmare({ show: true })
var dt = (new Date()).getTime();
var filename = `/tmp/image-${dt}.png`;
nightmare
.goto('http://yahoo.com')
.type('form[action*="/search"] [name=p]', 'github nightmare')
.click('form[action*="/search"] [type=submit]')
.wait('#main')
.evaluate(function () {
return document.querySelector('#main .searchCenterMiddle li a').href
})
.screenshot(filename)
.end()
.then(function (result) {
console.log(`Screenshot was saved to filename ${filename}`);
console.log('Result: ', result)
})
.catch(function (error) {
console.error('Search failed:', error);
});
[nightmare-test]$ xvfb-run -a --server-args="-screen 0 1366x768x24" node example-screenshot.js
Screenshot was saved to filename /tmp/image-1488912962584.png
Result: undefined
# Let's check size of the screenshot
$ ls -lh /tmp/*.png
-rw-rw-r-- 1 ec2-user ec2-user 87K Mar 7 18:56 /tmp/image-1488912962584.png
If you still want to look at the actual image, you find many ways to get it to you: scp, upload to S3, send to email.
I will show the simplest way to email attachment from linux box using mutt command:
sudo yum install -y mutt # let's install mutt mail client
echo "Sending screenshot" | mutt -s "Nightmare screenshot" [email protected] -a /tmp/image-1488912962584.png
# Be sure to check your Spam folder for this message =)
And here's the reward:
@rinogo Totally checking out your guide. Good to see a friendly face :) Hope all os well.