Umar Butler PRO

Hey @elsatch . Eventually, yes. But it'll take a lot more work to get right!

Right now, we've got a fairly diverse and representative benchmark of English-language legal tasks. Adding one Chinese task here and one German task there would immediately skew results towards whichever model is the most 'multilingual' and not necessarily the best at generalized legal retrieval.

To control for that, we'd essentially need to create multiple copies of MLEB, each for a different language, which would ensure that, for example, a model can't be regarded as the best at Arabic legal retrieval simply because it does well at two specific contract-focused datasets.

Building MLEB took quite a bit of work and effort. A lot of time was spent on curation rather than creation. There were many datasets that didn't make the cut, either due to being of low quality or being too trivial.

I have a law degree and previously led AI at the Australian Attorney-General's Department, so I've been able to draw on that knowledge and experience to understand what's good and what isn't.

We'd need to have others from different cultures who have similar backgrounds to be able to help with ensuring that their language's data was also of high quality and trustworthy provenance.

Thanks @tomaarsen ! We're big fans of your work, some of which helped build Kanon 2 Embedder!

The vast majority of our training data was out-of-distribution (relative to MLEB), however, there was definitely some overlap, which is hard to avoid when training only on domain data. We took extra care though to ensure no overlap with any test sets, including test sets not included in MLEB (e.g., AILA (which was of relatively low quality anyway), LegalBench, etc.).

Ultimately, the most important test set is your own real-world data, hence why we encourage people to try Kanon 2 Embedder for themselves and see if it improves performance 😊