Error from ISpark to run Spooky Stuff

ispooky_issues.md

Error Related to Spooky Stuff and ISpooky

Error with ISpark

We are not be able to run ISpark for spooky stuff. I can run notebook on the previous version of spooky stuff. However, for the newer version, we are not sure it is a problem of .jar file or not. This is error that we got:

Exception in thread "RequestsEventLoop" java.lang.RuntimeException: SpookyContext failed to be imported
at org.tribbloid.ispark.SpookyInterpreter.initializeSpark(SpookyInterpreter.scala:23)
at org.tribbloid.ispark.SparkInterpreter.<init>(SparkInterpreter.scala:57)
at org.tribbloid.ispark.SQLInterpreter.<init>(SQLInterpreter.scala:12)
at org.tribbloid.ispark.SpookyInterpreter.<init>(SpookyInterpreter.scala:6)
at org.tribbloid.ispark.Main.interpreter$lzycompute(Main.scala:41)
at org.tribbloid.ispark.Main.interpreter(Main.scala:38)
at org.tribbloid.ispark.CompleteHandler.apply(Handlers.scala:233)
at org.tribbloid.ispark.Main$EventLoop$$anonfun$run$2.apply(Main.scala:113)
at org.tribbloid.ispark.Main$EventLoop$$anonfun$run$2.apply(Main.scala:110)
at scala.Option.foreach(Option.scala:236)
at org.tribbloid.ispark.Main$EventLoop.run(Main.scala:110)

Here is the gist that I provided to ipython_config.py where this worked with the older version of Spooky Stuff

import os
c = get_config()

SPARK_HOME = os.environ['SPARK_HOME']
MASTER = 'local[4]'
c.KernelManager.kernel_cmd = [SPARK_HOME+"/bin/spark-submit",
 "--master", MASTER,
 "--class", "org.tribbloid.ispark.Main",
 "--executor-memory", "2G", #ISpark driver takes more memory than most other Spark drivers
 "--jars", "/home/ubuntu/spookystuff/shell/target/scala-2.10/spookystuff-shell-assembly-0.3.0-SNAPSHOT.jar",
 "/home/ubuntu/ISpark/core/target/ispark-core-assembly-0.1.0-SNAPSHOT.jar",
 "--profile", "{connection_file}",
 "--interp", "Spooky",
 "--parent"]

c.NotebookApp.ip = '*' # only add this line if you want IPython-notebook being open to the public
c.NotebookApp.open_browser = False # only add this line if you want to suppress opening a browser after IPython-notebook initialization 

Error with #{_} in example

This is error related to example (Youtube.scala example). I am not able to use the same #{_} to replace word in the new version of Spooky Stuff provided in the example. The error is given below:

org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Unresolved attributes: 'title, tree:
Project [_#0,'title]
 SparkLogicalPlan (ExistingRdd [_#0], MappedRDD[22] at map at JsonRDD.scala:38)

	at org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$$anonfun$apply$1.applyOrElse(Analyzer.scala:72)
	at org.apache.spark.sql.catalyst.analysis.Analyzer$CheckResolution$$anonfun$apply$1.applyOrElse(Analyzer.scala:70)

All problem solved see ISpooky Github