Youngjoon Jang's picture

Youngjoon Jang

yjoonjang

·

https://yjoonjang.github.io/

AI & ML interests

Information Retrieval (IR), Retrieval-Augmented Generation (RAG)

Recent Activity

new activity 14 days ago

yjoonjang/colbert-ko-v1.0:Produce ColBERT-KO Evaluation Results

reacted to Norod78's post with 👍 16 days ago

Multilingual Tokenization Showdown Analyzing 12 LLM Tokenizers Across 204 Languages. First, I've created a dataset with Wikipedia's "Cat" article text in 272 languages: https://huggingface.co/datasets/Norod78/WikiCat-Multilingual For each language entry with at least 100 words, I tokenized the text using 12 tokenizers and calculated the "Characters per token" ratio and "Word per token" ratio. The higher this ratio is, the more information each token represents on average for that language (and perhaps allowing the llm to potentially learn more per-parameter if trained on a dataset of that language). You can see a slideshow summary of the results here: https://norod.github.io/wikicat-tokenizer-eval/tokenizer-slideshow.html I hope I interpreted the results correctly, I've made the code available on GitHub so you can re-create the raw results jsonl with this repo: https://github.com/Norod/wikicat-tokenizer-eval Post on X: https://x.com/Norod78/status/1984366900550266999

liked a dataset 29 days ago

whybe-choi/trec-dl-2020

View all activity

Organizations

upvoted an article about 2 months ago

Article

Introducing RTEB: A New Standard for Retrieval Evaluation

Oct 1

•

126

upvoted a collection 3 months ago

Embedding-Preview

5 items • Updated Sep 17 • 5

upvoted a paper 3 months ago

SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens

Paper • 2508.05305 • Published Aug 7 • 46

upvoted a paper 6 months ago

Crosslingual Reasoning through Test-Time Scaling

Paper • 2505.05408 • Published May 8 • 8

upvoted a collection 7 months ago

HyperCLOVA X SEED

HyperCLOVA X SEED is NAVER's lightweight open-source lineup with a strong focus on Korean language performance • 4 items • Updated Jul 22 • 28

upvoted a paper 7 months ago

MIEB: Massive Image Embedding Benchmark

Paper • 2504.10471 • Published Apr 14 • 20

upvoted a collection 8 months ago

EXAONE-Deep

EXAONE reasoning model series of 2.4B, 7.8B, and 32B, optimized for reasoning tasks including math and coding • 10 items • Updated Jul 7 • 94

upvoted a collection about 1 year ago

Magpie-Llama3.1 Datasets

Dataset built with Meta Llama 3.1 70B. • 6 items • Updated Jan 13 • 4