Makoto YUI myui

$ diff build.xml build.xml.orig
41,42c41,42
<       <echo message="Use Hadoop 2.6.0 by default" />
<       <property name="hadoopversion" value="260" />
---
>       <echo message="Use Hadoop 2.x by default" />
>       <property name="hadoopversion" value="200" />
188,201d187
<

Hivemall provides a batch learning scheme that builds prediction models on Apache Hadoop. The learning process itself is a batch process; however, an online/real-time prediction can be achieved by carrying a prediction on a transactional relational DBMS.

In this article, we explain how to run a real-time prediction using a relational DBMS. We assume that you have already run the a9a binary classification task.

Online/Offline Matrix of Machine Learning

The following table shows the type matrix of machine learning schemes and applications.

First of all, make sure that your Treasure Data cluster is HDP2, not CDH4. Matrix Factorization is only supported in the up-to-date HDP2 cluster. HDP2 is allocated for users who signed Treasure Data after Feb 2015. CDH4 is allcoated for the others.

NOTE: please ask our customer support to use HDP2 if you get an error.

Data preparation

Download ml-20m.zip and unzip it.

	Vertex failed, vertexName=Map 20, vertexId=vertex_1424704867400_0022_3_49, diagnostics=[Task failed, taskId=task_1424704867400_0022_3_49_000000, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"gid":1,"userid":4422,"movieid":1213,"rating":5}
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:186)
	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:138)
	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:324)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
	at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.S

	/*
	* Hivemall: Hive scalable Machine Learning Library
	*
	* Copyright (C) 2013-2014
	* National Institute of Advanced Industrial Science and Technology (AIST)
	* Registration Number: H25PRO-1520
	*
	* This library is free software; you can redistribute it and/or
	* modify it under the terms of the GNU Lesser General Public
	* License as published by the Free Software Foundation.

	def time[R](block: => R): R = {
	val t0 = System.nanoTime()
	val result = block
	val t1 = System.nanoTime()
	println("Elapsed time: " + (t1 - t0) + "ns")
	result
	}

	val result = time { 1 to 1000 sum }

	#!/bin/bash

	# Licensed to the Apache Software Foundation (ASF) under one or more
	# contributor license agreements. See the NOTICE file distributed with
	# this work for additional information regarding copyright ownership.
	# The ASF licenses this file to You under the Apache License, Version 2.0
	# (the "License"); you may not use this file except in compliance with
	# the License. You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0

	#!/bin/bash

	# Licensed to the Apache Software Foundation (ASF) under one or more
	# contributor license agreements. See the NOTICE file distributed with
	# this work for additional information regarding copyright ownership.
	# The ASF licenses this file to You under the Apache License, Version 2.0
	# (the "License"); you may not use this file except in compliance with
	# the License. You may obtain a copy of the License at
	#
	# http://www.apache.org/licenses/LICENSE-2.0

	#cloud-config

	hostname: dcXX
	fqdn: dcXX.ec2.internal

	mounts:
	- [ xvdb, /mnt/disk1, "auto", "defaults,nobootwait,comment=cloudconfig", 0, 2]
	- [ xvdc, /mnt/disk2, "auto", "defaults,nobootwait,comment=cloudconfig", 0, 2]

	runcmd:

	HivemallのMatrix Factorization学習のパラメタの説明です。
	http://qiita.com/myui/items/dccb4f58799f080e24ab#%E3%83%90%E3%82%A4%E3%82%A2%E3%82%B9%E3%82%92%E8%80%83%E6%85%AE%E3%81%97%E3%81%9F-matrix-factorization

	factor, mu, iterations以外は通常指定不要です。指定順序は関係ありません。
	etaは場合によっては指定したほうがよいケースもあります。

	1) "-factor 10"
	The number of latent factor [default: 10]
	潜在変数の数