AI & softwareResearch findings
Taxonomy and Evaluation of Embedding-Based Diversity Measures
Original published 2026.10.18 07:09 KSTAdded to Blackdeck 2026.10.20 11:35 KST
What has changed?
A systematic taxonomy of 21 embedding-based diversity measures is proposed, grouped into four computational families.
Key points
- Text corpus diversity has direct implications for model generalization and data curation.
- Existing embedding-based diversity measures vary widely in computation and yield inconsistent results.
- EmbDivBench evaluates diversity across variety, balance, and disparity using both simulated and real text data.
Connect it to your work
Graph Entropy outperforms Mean Pairwise Distance across all three diversity components in empirical evaluation.
What remains unproven
The abstract reports simulation results; real-world performance is not established by this input.
Originals used for this note
Links open the original publisher in a new tab.