NuPIC OPF swarming - is really best model chosen?

swarming_models

####>> UPDATED JOB RESULTS: 
{   u'bestModel': 104195,
    u'bestValue': 4.43,
    u'metrics': {   u"multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg": 4.43,
                    u"multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.46635898904319556,
                    u"prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg": 3.752,
                    u"prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.39498395640859363},
    u'saved': True} (elapsed time: 31110.6 secs)
##>> UPDATED STATUS: 
Completed generation #2 of swarm 'modelParams|sensorParams|encoders|ecg' with a best errScore of 4.601. Matured swarm(s): [u'modelParams|sensorParams|encoders|ecg']
####>> UPDATED JOB RESULTS: 
{   u'absoluteFieldContributions': {   u'ecg': 0.0},
    u'bestModel': 104195,
    u'bestValue': 4.43,
    u'fieldContributions': {   u'ecg': 0.0},
    u'metrics': {   u"multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg": 4.43,
                    u"multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.46635898904319556,
                    u"prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg": 3.752,
                    u"prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg": 0.39498395640859363},
    u'saved': True} (elapsed time: 31111.6 secs)
Evaluated 20 models
HyperSearch finished!
Worker completion message: Exiting because we've evaluated all possible field combinations

Results from all experiments:
----------------------------------------------------------------
[0] Experiment _NupicModelInfo(jobID=2460, modelID=104149, status=completed, completionReason=eof, updateCounter=44, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.482 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.47183318033670485
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[1] Experiment _NupicModelInfo(jobID=2460, modelID=104153, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.02.modelParams|tpParams|minThreshold_9.modelParams|tpParams|activationThreshold_12.modelParams|tpParams|pamLength_10.modelParams|sensorParams|encoders|ecg:w_51):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      6.261 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.659113686320417
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[2] Experiment _NupicModelInfo(jobID=2460, modelID=104171, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_12.modelParams|tpParams|activationThreshold_16.modelParams|tpParams|pamLength_50.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.599 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4841501107471008
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[3] Experiment _NupicModelInfo(jobID=2460, modelID=104182, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.5.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_13.modelParams|tpParams|pamLength_100.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.623 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4866766605748743
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[4] Experiment _NupicModelInfo(jobID=2460, modelID=104183, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_360.modelParams|sensorParams|encoders|ecg:w_51):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.592 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4834132003806668
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[5] Experiment _NupicModelInfo(jobID=2460, modelID=104184, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.599 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4841501107471008
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[6] Experiment _NupicModelInfo(jobID=2460, modelID=104185, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.655 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4900453936785723
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[7] Experiment _NupicModelInfo(jobID=2460, modelID=104186, status=completed, completionReason=eof, updateCounter=27, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.447 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.46814862850453515
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[8] Experiment _NupicModelInfo(jobID=2460, modelID=104187, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_13.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.454 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4688855388709691
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[9] Experiment _NupicModelInfo(jobID=2460, modelID=104188, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.604 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4846764752945536
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[10] Experiment _NupicModelInfo(jobID=2460, modelID=104189, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.461 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.46962244923740304
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[11] Experiment _NupicModelInfo(jobID=2460, modelID=104190, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_16.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.601 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4843606565660819
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[12] Experiment _NupicModelInfo(jobID=2460, modelID=104191, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.621 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.4864661147558932
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[13] Experiment _NupicModelInfo(jobID=2460, modelID=104192, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      5.141 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.5412080276909861
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[14] Experiment _NupicModelInfo(jobID=2460, modelID=104193, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      5.139 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.540997481872005
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[15] Experiment _NupicModelInfo(jobID=2460, modelID=104194, status=completed, completionReason=eof, updateCounter=25, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_14.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      5.13 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.5400500256865899
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[16] Experiment _NupicModelInfo(jobID=2460, modelID=104195, status=completed, completionReason=eof, updateCounter=27, numRecords=2000)
(modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.43 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.46635898904319556
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      3.752
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.39498395640859363

[17] Experiment _NupicModelInfo(jobID=2460, modelID=104196, status=completed, completionReason=killed, updateCounter=14, numRecords=1050)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_12.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_1800.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      8.725 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.9135867362912252
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      4.422
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.46302355849625193

[18] Experiment _NupicModelInfo(jobID=2460, modelID=104197, status=completed, completionReason=killed, updateCounter=12, numRecords=800)
(modelParams|sensorParams|encoders|ecg:resolution_0.5.modelParams|tpParams|minThreshold_11.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      8.774718397997496 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.9194139146103015
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      4.264080100125156
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.4467897884862783

[19] Experiment _NupicModelInfo(jobID=2460, modelID=104198, status=completed, completionReason=killed, updateCounter=5, numRecords=150)
(modelParams|sensorParams|encoders|ecg:resolution_0.1.modelParams|tpParams|minThreshold_12.modelParams|tpParams|activationThreshold_16.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_91):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      2.872483221476511 (*)
  multiStepBestPredictions:multiStep:errorMetric='altMAPE':steps=1:window=1000:field=ecg:  0.3045092989171423
  prediction:trivial:errorMetric='aae':steps=1:window=1000:field=ecg:                      2.1476510067114094
  prediction:trivial:errorMetric='altMAPE':steps=1:window=1000:field=ecg:                  0.22767050386328386

--------------------------------------------------------------
20 experiments total (all completed successfully).

WaitingToStart: 0
Running: 0
Completed: 20
    ran to EOF: 17
    ran to stop signal: 0
    were orphaned: 0
    killed off: 3
    failed: 0

Field Contributions:
{   u'ecg': 0.0}

Best results on the optimization metric multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg (maximize=False):
[16] Experiment _NupicModelInfo(jobID=2460, modelID=104195, status=completed, completionReason=eof, updateCounter=27, numRecords=2000) (modelParams|sensorParams|encoders|ecg:resolution_0.2.modelParams|tpParams|minThreshold_10.modelParams|tpParams|activationThreshold_15.modelParams|tpParams|pamLength_5.modelParams|sensorParams|encoders|ecg:w_31):
  multiStepBestPredictions:multiStep:errorMetric='aae':steps=1:window=1000:field=ecg:      4.43

Total number of Records processed: 36000

Total wall time for all models: 124434

Generating description files for top 1 models...
Generating description file for model 104195 at /home/mmm/devel/ecg/nupic.biodat/ECG/MIT-BIH-Arrhythmia/model/swarm/model_0
Generating model params file...

@rhyolight why was this model killed? https://gist.github.com/breznak/fb12394015fb4108a069#file-swarming_models-L160
It had best so far performance!