Cross-Discourse and Multilingual Exploration of Textual Corpora with the DualNeighbors Algorithm
Table 3: Connectivity metrics for similarity graphs. All examples relate each item to twelve neighbors, with Nw word neighbors and Ne embedding neighbors. For comparison, we show the results using both query replacement (as described in the DualNeighbors algorithm) and with the query expansion method suggested in the papers discussed in Section 2.2. The metrics give the (undirected) spectral gap λ2, the proportion of directed pairs of items that are unconnected across directed edges (u.c.), the average distance (dist) between connected pairs of items, the 90th percentile of the in-degree (din0.9), and the 10th percentile of the number of neighbors within three links (ego(3)0.1).
['[EMPTY]', '[ITALIC] Nw', '[ITALIC] Ne', 'FSA-OWI  [ITALIC] λ2', 'FSA-OWI u.c.', 'FSA-OWI dist', 'FSA-OWI  [ITALIC] din0.9', 'FSA-OWI ego0.1', 'Twitter  [ITALIC] λ2', 'Twitter u.c.', 'Twitter dist', 'Twitter  [ITALIC] din0.9', 'Twitter ego(3)0.1']
The DualNeighbors algorithm features two crucial differences compared to other word-embedding based query expansion techniques. Splitting the search explicitly into two types of neighbors allows for a better balance between the connectivity and diversity of the recommended documents. Also, replacing the document with its closest word embeddings, rather than augmenting as other approaches have done, significantly improves the diversity of the recommended documents. Additionally, by varying the number of neighbors displayed by each method, users can manually adjust the balance between diversity and relevance in the results. All of the examples use a total of 12 recommendations for consistency. Generally, we see that adding more edges from the (query expansion) word embedding matrix produces a network with a larger algebraic connectivity, lower average distance between document pairs, and larger third-degree ego scores. The distribution of in-degrees becomes increasingly variable, however, as more edges get mapped to a small set of hubs (documents linked to from a very large number of other documents). These two effects combine so that the most connected network using both corpora have 10 edges from word similarities and 2 edges from the word embedding logic. Generally, including at least one word embedding edges makes the network significantly more connected. The hubness of the network slowly becomes an issue as the proportion of embedding edges grows relative to the total number of edges. To illustrate the importance of using query replacement in the word embedding neighbor function, the table also compares our approach (query replacement) to that of query expansion. Therefore, this modification serves as an important contribution and distinguishing feature of our approach.