wpm
/ spark_parallel_boost.py
Last active
December 3, 2018 02:56
A simple example of how to integrate the Spark parallel computing framework and the scikit-learn machine learning toolkit. This script randomly generates test and train data sets, trains an ensemble of decision trees using boosting, and applies the ensemble to the test set. The ensemble training is done in parallel.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from pyspark import SparkContext | |
| import numpy as np | |
| from sklearn.cross_validation import train_test_split, Bootstrap | |
| from sklearn.datasets import make_classification | |
| from sklearn.metrics import accuracy_score | |
| from sklearn.tree import DecisionTreeClassifier | |
| def run(sc): |