Installation document for Spooky Stuff

spookystuff_install.md

Installing Spooky Stuff

Here I list some installation approach that I use to install Spooky Stuff

Launch amazon instance

Install some useful packages

sudo apt-get update
sudo apt-get install emacs
sudo apt-get install git
sudo apt-get install maven

Download anaconda for ubuntu and install

wget http://09c8d0b2229f813c1b93-c95ac804525aac4b6dba79b00b39d1d3.r79.cf1.rackcdn.com/Anaconda-2.1.0-Linux-x86_64.sh
bash Anaconda-2.1.0-Linux-x86.sh

and add following path for anaconda export PATH=$HOME/anaconda/bin:$PATH to .bash_profile

Download java

using sudo apt-get install openjdk-7-jdk and add java path to .bash_profile as follow export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

Download spark

We can download spark from https://spark.apache.org/downloads.html and un-tar the file

wget http://d3kbcqa49mib13.cloudfront.net/spark-1.1.1-bin-hadoop1.tgz
tar -xf spark-1.1.1-bin-hadoop1.tgz

and add path to .bash_profile as follow export SPARK_HOME=/home/ubuntu/spark-1.1.1-bin-hadoop1

Download phantomjs

Here is a link to download phantomjs: http://phantomjs.org/download.html Then we un-tar the file.

wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.8-linux-x86_64.tar.bz2
tar -xf phantomjs-1.9.8-linux-x86_64.tar.bz2

then bin/phantomjs is ready to use and we will copy it to /usr/lib

sudo cp phantomjs-1.9.8-linux-x86_64/bin/phantomjs /usr/lib/

after that, adding a new path to .bash_profile as export PATH=${PATH}:/usr/lib

Download Spooky Stuff

We can download spookystuff from github use following line

git clone https://github.com/tribbloid/spookystuff
cd spookystuff
MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" mvn package -DskipTests=true

Now we change to ./mvn-package.sh instead.

Run Spooky Stuff

./bin/spooky-shell.sh

Issues

We found error in test section when we install Spooky Stuff as follow

revert proxy setting for https wget *** FAILED ***
  org.tribbloid.spookystuff.ActionException: +>Wget('https://www.astrill.com/what-is-my-ip-address.php')
driver not initialize, snapshot/screenshot not available
Caused by: java.net.SocketException: Connection refused
  at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:436)
  at java.net.Socket.connect(Socket.java:579)
  at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:250)
  at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
  at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
  at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
  at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
  at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
  at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
  at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
  ...

so we need to comment the maven test section in pom.xml file

<!--plugin>
  <groupId>org.scalatest</groupId> 
    <artifactId>scalatest-maven-plugin</artifactId> 
    <version>1.0</version> 
    <executions>
      <execution>
        <id>test</id> 
        <goals>
          <goal>test</goal> 
        </goals>
      </execution>
    </executions> 
</plugin-->

This is a link to spookystuff TestExport.scala file.

Remove factory in shell section

This following package does not exist anymore

spookystuff/core/src/main/scala/org/tribbloid/spookystuff/factory

where you include in spookystuff/shell/src/main/scala/org/tribbloid/spookystuff/repl/SpookyILoop.scala. Therefore, it gives rise to this error when I run spooky shell as follows:

console>:16: error: object factory is not a member of package org.tribbloid.spookystuff
import org.tribbloid.spookystuff.factory.driver._

According to the LinkedIn example, it seems that that import should be replaced by

import org.tribbloid.spookystuff.dsl._

After I added this line, everything works now

We solve both issue stated in Git (see Sppoky Stuff repo)