Makoto YUI myui

This article introduce how to find outliers using Local Outlier Detection (LOF) on Hivemall.

Data Preparation

create database lof;
use lof;

create external table hundred_balls (
  rowid int,

First of all, make sure that your Treasure Data cluster is HDP2, not CDH4. Matrix Factorization is only supported in the up-to-date HDP2 cluster. HDP2 is allocated for users who signed Treasure Data after Feb 2015. CDH4 is allcoated for the others.

NOTE: please ask our customer support to use HDP2 if you get an error.

Data preparation

Download ml-20m.zip and unzip it.

	_______1_______
	___2_______2___
	_4___4___4___4_
	8_8_8_8_8_8_8_8

	/*
	* Hivemall: Hive scalable Machine Learning Library
	*
	* Copyright (C) 2015 Makoto YUI
	*
	* Licensed under the Apache License, Version 2.0 (the "License");
	* you may not use this file except in compliance with the License.
	* You may obtain a copy of the License at
	*
	* http://www.apache.org/licenses/LICENSE-2.0

	set hivevar:k=11;

	create table similarities
	as
	WITH test_rnd as (
	select
	rand(31) as rnd,
	id,
	features
	from

	/*
	* Hivemall: Hive scalable Machine Learning Library
	*
	* Copyright (C) 2015 Makoto YUI
	* Copyright (C) 2013-2015 National Institute of Advanced Industrial Science and Technology (AIST)
	*
	* Licensed under the Apache License, Version 2.0 (the "License");
	* you may not use this file except in compliance with the License.
	* You may obtain a copy of the License at
	*

	create table similarities
	as
	WITH test_rnd as (
	select
	rand(31) as rnd,
	id,
	features
	from
	test_hivemall
	),

	create table similarities
	as
	SELECT
	each_top_k(
	10, t2.id, angular_similarity(t2.features, t1.features),
	t2.id,
	t1.id,
	t1.y
	) as (rank, similarity, base_id, neighbor_id, y)
	FROM

	HivemallのMatrix Factorization学習のパラメタの説明です。
	http://qiita.com/myui/items/dccb4f58799f080e24ab#%E3%83%90%E3%82%A4%E3%82%A2%E3%82%B9%E3%82%92%E8%80%83%E6%85%AE%E3%81%97%E3%81%9F-matrix-factorization

	factor, mu, iterations以外は通常指定不要です。指定順序は関係ありません。
	etaは場合によっては指定したほうがよいケースもあります。

	1) "-factor 10"
	The number of latent factor [default: 10]
	潜在変数の数