eliasah

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *

eliasah

Info

This guide sets up a non-clustered Nutch crawler, which stores its data via HBase. We will not learn how to setup Hadoop et al., but just the bare minimum to crawl and index websites on a single machine.

Terms

• Nutch - the crawler (fetches and parses websites)
• HBase - filesystem storage for Nutch (Hadoop component, basically)

eliasah

/*
This example uses Scala.  Please see the MLlib documentation for a Java example.

Try running this code in the Spark shell.  It may produce different topics each time (since LDA includes some randomization), but it should give topics similar to those listed above.

This example is paired with a blog post on LDA in Spark: http://databricks.com/blog
Spark: http://spark.apache.org/
*/

import scala.collection.mutable

eliasah

# delete all data
curl -XDELETE localhost:9200/test

# create an index and define specific french stop_words
curl -XPUT localhost:9200/test -d '{
    "settings" : {
        "index" : {
            "analysis" : {
                "analyzer" : {
                    "french" : {

eliasah

// derived from http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm

function map() {
    emit(1, // Or put a GROUP BY key here
         {sum: this.value, // the field you want stats for
          min: this.value,
          max: this.value,
          count:1,
          diff: 0, // M2,n:  sum((val-mean)^2)
    });

eliasah

Kibana 3 against ElasticSearch 1.4 throws an **Connection Failed** screen. The error text says to set `http.cors.allow-origin`, but it misses out the important `http.cors.enabled: true`

Working config: 

    $ grep cors elasticsearch-1.4.0.Beta1/config/elasticsearch.yml
    http.cors.allow-origin: "/.*/"
    http.cors.enabled: true

* [Ref](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-http.html)
* [Ref](http://elasticsearch-users.115913.n3.nabble.com/Kibana-upgrade-trouble-nor-4-0BETA1-neither-3-11-work-now-td4064625.html)

eliasah

wget http://apt.puppetlabs.com/puppetlabs-release-precise.deb
sudo dpkg -i puppetlabs-release-precise.deb
sudo apt-get update

eliasah

#!/bin/sh
# This script installs Scala 2.10.3 with SBT 0.13 on Ubuntu 12.04

wget http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
tar zxf scala-2.10.3.tgz sudo mv scala-2.10.3 /usr/local/share/scala

sudo ln -s /usr/local/share/scala/bin/scala /usr/bin/scala
sudo ln -s /usr/local/share/scala/bin/scalac /usr/bin/scalac
sudo ln -s /usr/local/share/scala/bin/fsc /usr/bin/fsc
sudo ln -s /usr/local/share/scala/bin/scaladoc /usr/bin/scaladoc 

eliasah

#!/bin/bash

ES='http://localhost:9200'
ESIDX='test3'
ESTYPE='test'

curl -XDELETE $ES/$ESIDX

curl -XPUT $ES/$ESIDX/ -d '{
	"settings" : {

eliasah

# Find the MD5 sum of the current directory
find . -type f | grep -v "^./.git" | xargs md5 | md5