Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

README.md

Gil Levi and Tal Hassner, Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns

Convolutional neural networks for emotion classification from facial images as described in the following work:

Gil Levi and Tal Hassner, Emotion Recognition in the Wild via Convolutional Neural Networks and Mapped Binary Patterns, Proc. ACM International Conference on Multimodal Interaction (ICMI), Seattle, Nov. 2015

Project page: http://www.openu.ac.il/home/hassner/projects/cnn_emotions/

If you find our models useful, please add suitable reference to our paper in your work.

gist_id: 54aee1b8b0397721aa4b

Emotion Classification CNN - RGB

caffemodel: VGG_S_rgb/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSNVYtWnNPMU1TOGM

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSTEZpN3ItN1c0LUE

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSOFp4Ri1ITzZuUkk

Emotion Classification CNN - LBP

caffemodel: VGG_S_lbp/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XScTFlWkszazA0b0E

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSNXBRVnlZS0FsdzQ

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSckxVTmN5YUlfUGM

Emotion Classification CNN - Cyclic LBP

caffemodel: VGG_S_cyclic_lbp/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSNUFVSDZydS0yc2c

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSWXJ0UnFHaHROQUU

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSbHdoU1N0LWtFRG8

Emotion Classification CNN - Cyclic LBP-5

caffemodel: VGG_S_cyclic_lbp_5/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSYjVxbGJZYVdveWM

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSa0pXXzJ1ZGNPYTg

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSdndLMnpDMWItLTA

Emotion Classification CNN - Cyclic LBP-10

caffemodel: VGG_S_cyclic_lbp_10/EmotiW_VGG_S.caffemodel

caffemodel_url: https://drive.google.com/open?id=0BydFau0VP3XSOWlZekZXWGdySEU

mean_file_proto: https://drive.google.com/open?id=0BydFau0VP3XSenRYa2Q1S3E0Q2s

deploy_file: https://drive.google.com/open?id=0BydFau0VP3XSMGkxMTgtU0U1dVU

---

Copyright 2015, Gil Levi and Tal Hassner

The SOFTWARE provided in this page is provided "as is", without any guarantee made as to its suitability or fitness for any particular use. It may contain bugs, so use of this tool is at your own risk. We take no responsibility for any damage of any sort that may unintentionally be caused through its use.

train_val.prototxt

name: "CaffeNet"
layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label"
  data_param {
    source: "/home/ubuntu/EmotiW/lmdb/train_lmdb"
    backend: LMDB
    batch_size: 30
  }
  transform_param {
    crop_size: 224
    mean_file: "/home/ubuntu/EmotiW/mean_image/mean.binaryproto"
    mirror: true
  }
  include: { phase: TRAIN }
}
layers {
  name: "data"
  type: DATA
  top: "data"
  top: "label"
  data_param {
    source: "/home/ubuntu/EmotiW/lmdb/val_lmdb"
    backend: LMDB
    batch_size: 20
  }
  transform_param {
    crop_size: 224
    mean_file: "/home/ubuntu/EmotiW/mean_image/mean.binaryproto"
    mirror: false
  }
  include: { phase: TEST }
}
layers {
  name: "conv1"
  type: CONVOLUTION
  bottom: "data"
  top: "conv1"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 96
    kernel_size: 7
    stride: 2
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "relu1"
  type: RELU
  bottom: "conv1"
  top: "conv1"
}
layers {
  name: "norm1"
  type: LRN
  bottom: "conv1"
  top: "norm1"
  lrn_param {
    local_size: 5
    alpha: 0.0005
    beta: 0.75
  } 
} 
layers {
  name: "pool1"
  type: POOLING
  bottom: "norm1"
  top: "pool1"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 3
  }
}
layers {
  name: "conv2"
  type: CONVOLUTION
  bottom: "pool1"
  top: "conv2"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 256
    pad: 2
    kernel_size: 5
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu2"
  type: RELU
  bottom: "conv2"
  top: "conv2"
}
layers {
  name: "pool2"
  type: POOLING
  bottom: "conv2"
  top: "pool2"
  pooling_param {
    pool: MAX
    kernel_size: 2
    stride: 2
  }
}
layers {
  name: "conv3"
  type: CONVOLUTION
  bottom: "pool2"
  top: "conv3"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "relu3"
  type: RELU
  bottom: "conv3"
  top: "conv3"
}
layers {
  name: "conv4"
  type: CONVOLUTION
  bottom: "conv3"
  top: "conv4"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu4"
  type: RELU
  bottom: "conv4"
  top: "conv4"
}
layers {
  name: "conv5"
  type: CONVOLUTION
  bottom: "conv4"
  top: "conv5"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  convolution_param {
    num_output: 512
    pad: 1
    kernel_size: 3
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu5"
  type: RELU
  bottom: "conv5"
  top: "conv5"
}
layers {
  name: "pool5"
  type: POOLING
  bottom: "conv5"
  top: "pool5"
  pooling_param {
    pool: MAX
    kernel_size: 3
    stride: 3
  }
}
layers {
  name: "fc6"
  type: INNER_PRODUCT
  bottom: "pool5"
  top: "fc6"
  blobs_lr: 1
  blobs_lr: 2
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4048
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu6"
  type: RELU
  bottom: "fc6"
  top: "fc6"
}
layers {
  name: "drop6"
  type: DROPOUT
  bottom: "fc6"
  top: "fc6"
  dropout_param {
    dropout_ratio: 0.7
  }
}
layers {
  name: "fc7"
  type: INNER_PRODUCT
  bottom: "fc6"
  top: "fc7"
  blobs_lr: 9
  blobs_lr: 18
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 4048
    weight_filler {
      type: "gaussian"
      std: 0.005
    }
    bias_filler {
      type: "constant"
      value: 1
    }
  }
}
layers {
  name: "relu7"
  type: RELU
  bottom: "fc7"
  top: "fc7"
}
layers {
  name: "drop7"
  type: DROPOUT
  bottom: "fc7"
  top: "fc7"
  dropout_param {
    dropout_ratio: 0.7
  }
}
layers {
  name: "fc8_cat"
  type: INNER_PRODUCT
  bottom: "fc7"
  top: "fc8"
  blobs_lr: 12
  blobs_lr: 24
  weight_decay: 1
  weight_decay: 0
  inner_product_param {
    num_output: 7
    weight_filler {
      type: "gaussian"
      std: 0.01
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}
layers {
  name: "accuracy"
  type: ACCURACY
  bottom: "fc8"
  bottom: "label"
  top: "accuracy"
  include: { phase: TEST }
}
layers {
  name: "loss"
  type: SOFTMAX_LOSS
  bottom: "fc8"
  bottom: "label"
  top: "loss"
}

Hi @LarryAndCode,

I have never tested my code on FER. Did you apply cropping prior to running the network?

Hi Gil

Im using the RGB model...and i keep getting the "Sad" emotion being chosen with about 60 to 80 percent almost all of the time...it never picks any other emotion...is there an update to this model that i am missing?
Thanks for the help

Steve

Hi Gil
Any chance i can get your image data so i can train my own model?...im having trouble converting some of your models to a mlmodel....thanks for the help.

Steve

Hi Gil, I wanted to use your model in my project, but I can't download the model, because Google says that access to the file is denied, there is some way to get to the model. I try dowload Emotion Classification CNN - RGB(https://drive.google.com/open?id=0BydFau0VP3XSNVYtWnNPMU1TOGM).