Linking emotions to behaviors through deep transfer learning
Table 4: Behavior binary classification accuracy in percentage for context-dependent behavior recognition model from emotion-embeddings. Bold numbers represent the best performing system.
['[EMPTY]', '[BOLD] Average', 'Acceptance', 'Blame', 'Positivity', 'Negativity', 'Sadness']
While E-BP_l model means the first l layers of the pretrained ER network are fixed and their output is used as the embedding E-BP for the subsequent system. As seen in the second column of the table, all of E-BP based models perform significantly better than the B-BP based model, which achieves an improvement of 8.57% on average and up to 16.78% for Negativity. An interesting experiment is what happens if we use a lower-ratio of emotion (out-of-domain) vs. behavior (couples-in-domain) data. What we observe is that the reduced learning taking place on emotional data requires the in-domain system to have prefer embeddings closer to the feature. Specifically Negativity performs equally well with layers 3 or 4 at 71.43%. Positivity performs best with layer 3 at 64.64%, Blame and Acceptance perform best with layer 2 at 71.07% and 72.86% respectively while Sadness performs best through layer 1 at 56.07%. In the reduced data case we observe that best performing layer is not consistently layer 4. (E-BP_4) is always the best performing layer, thus showing that more emotion data provides better ability of transfer learning. Based on E-BP-4 embeddings, the reduced context-dependent model employs 4 more CNN layers with optional local average pooling layers in between, and is followed by an adaptive max pooling layer and three fully connected layers to predict the session level label directly without sequential modules. We use PUR to sense the relative improvement from E-BP based context-dependent and E-BP based reduced context-dependent models respectively, to the baseline B-BP based context-dependent model. The larger value of PUR suggests the clear improvement of behavior prediction.