NLSC: Unrestricted Natural Language-based Service Composition through Sentence Embeddings
Table I: Confusion Matrix for 3 scenarios and 20 participants
['[EMPTY]', '[EMPTY]', '[BOLD] Predicted class  [BOLD] YES', '[BOLD] Predicted class  [BOLD] NO']
Results: For the Actual Class, we defined two values: 1) YES: user’s sentence is well structured, has meaning, can be understood, and should lead to the activation of a service and a specific method, and 2) NO: user’s sentence is ambiguous, or out of context, or incomprehensible, or should not lead to the activation of a service (method is not available or does not exist). For the Predicted Class, we defined two values: 1) YES: NLSC has correctly identified the method and service OR if user sentence was ambiguous, then it should ask user to re-phrase the sentence, and 2) NO : NLSC selected a wrong service and method OR id did not ask user to re-phrase the sentence. Since we have an uneven class distribution, that is, false positives and false negatives are very different, then Accuracy metric is not of too much help, thus we need to rely on F1-score instead due to it computes the weighted average of Precision and Recall. Generally speaking, results demonstrate a good performance since the recall, precision an F1-Score are above 0.5. However, it is worth noting that these values might vary significantly from one experiment to the other since they rely on human judgment, which is bias-prone.