SVM with SGD clarification for question http://apache-spark-user-list.1001560.n3.nabble.com/Need-help-with-SVM-tt27955.html

0.question_svm_with_sgd.md

Hello, I am using linear SVM to train my model and generate a line through my data. However my model always predicts 1 for all the feature examples. Here is my code:

print data_rdd.take(5) [LabeledPoint(1.0, [1.9643,4.5957]), LabeledPoint(1.0, [2.2753,3.8589]), LabeledPoint(1.0, [2.9781,4.5651]), LabeledPoint(1.0, [2.932,3.5519]), LabeledPoint(1.0, [3.5772,2.856])]

---

from pyspark.mllib.classification import SVMWithSGD from pyspark.mllib.linalg import Vectors from sklearn.svm import SVC data_rdd=x_df.map(lambda x:LabeledPoint(x[1],x[0]))

model = SVMWithSGD.train(data_rdd, iterations=1000,regParam=1)

X=x_df.map(lambda x:x[0]).collect() Y=x_df.map(lambda x:x[1]).collect()

---

pred=[] for i in X: pred.append(model.predict(i)) print pred

[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

1.svm_with_sgd.py

# code tested with pyspark 
# pyspark --packages com.databricks:spark-csv_2.10:1.5.0

from pyspark.ml.feature import VectorAssembler
from pyspark.mllib.classification import LabeledPoint
from pyspark.mllib.classification import SVMWithSGD

# read data
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true', inferschema='true').load('data.csv')

# prepare data
assembler = VectorAssembler(inputCols=["X", "Y"], outputCol="features")
data = assembler.transform(df)

# create RDD[LabeledPoint]
rdd = data.map(lambda row: LabeledPoint(row.label, row.features))

# Train SVM With SGD model
model = SVMWithSGD.train(rdd, iterations=1000,regParam=1.0,intercept=True,step=0.1)

# Create unlabled data
unlabeled_data = data.map(lambda x : x.features)

# Make prediction on unlabeled data and collect
model.predict(unlabeled_data).collect()
# [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]

2.data.csv

I didn't specify a threshold but you should be careful when splitting the data actually to perform stratified sampling so you don't up just with 1 labels in one split and 0s in the other.