Financial Event Extraction Using Wikipedia-Based Weak Supervision
Table 2: Number of articles and sentences in the News-2019 evaluation data.
['Company', 'Articles', 'Sentences']
In order to evaluate methods for detecting company-related events within news data, we compile a set of sentences from news articles. We assume that this set of articles provides a good coverage of the company’s events of interest during 2019. Ultimately we are interested in the ability to detect events in the target domain of news articles. To validate performance over this domain, we used sentences from News-2019 and cross-referenced them with company events from Wikipedia. For each event, we asked 3 annotators to mark all sentences from News-2019 which mention this event. In total, 26 of the Wikipedia events were mentioned in at least one sentence. So far our experiments considered only Wikipedia events. However, there are likely numerous company-related news events that are not necessarily mentioned in the company’s Wikipedia page. Thus, the question remains whether the Wikipedia-based models are able to detect such events as well. The guidelines were to determine whether a given sentence contains information which may have influence on the company’s stock price, as such events presumably deserve the attention of a finance analyst. The annotation process was composed of two stages. First, each sentence was annotated by two labelers. Then, the sentences on which there was disagreement between the labelers (21% of the sentences) were annotated by a third annotator. Average agreement between the initial two annotators was 0.45 (Cohen’s Kappa). Finally, we wanted to analyze the diversity of events captured by the two models. We sorted the remaining tokens by their frequency from highest to lowest, and computed the cumulative frequency as a function of the number of unique tokens. For example, 20% of the tokens are covered by the 36 and 19 most frequent tokens in Extended-wiki and SentiFM-binary, respectively. Moreover, despite their similar precision values, the population of events captured by the two models is quite different - the overlap between their top candidates is less than 10% (18 out of 200 examples). This observation suggests that the models are complementary, and that there is potential benefit to combining them.