The NLP Engine: A Universal Turing Machine for NLP
Table 1: Average entropy for different NLP tasks with correspondent dataset specified.
['Task', 'Avg Entropy']
As can be seen, MT is less complex than word prediction tasks, which is in line with our expectation: for MT, output tokens are predicated on source tokens. The input data provides additional information and lowers the degree of uncertainty: H(Y|X)≥H(Y) for any X and Y.