HARIN: A Novel Metric for Hierarchical Topic Model Assessment

Hierarchical topic modeling is a well-known technique for deriving comprehensive insights on a given dataset. However, it is challenging to choose the best-suited hierarchical topic model among many candidates, given that the model generally depends on the dataset under analysis. Moreover, even that chosen model typically produces an overwhelming number of leaf topics, making it hard to correctly interpret the result derived from the model. Although topic coherence is an often used metric to assess the quality of a model, coherence cannot reflect the unique characteristics of the hierarchical structure when applied to the model as well. To address these concerns, we propose a novel evaluation metric, HARIN (HierArchical haRmony INdex). The proposed HARIN metric effectively reflects the overall topic coherence, diversity, and similarity among parent-child and sibling layers in the produced hierarchy. We test the validity of HARIN by comparing it against competing metrics and human assessments of the quality of topic hierarchies from four popular models applied to five real-world datasets. Our experiments demonstrate that HARIN achieves approximately 85% accuracy in model ranking compared to human scores, surpassing the leading competing metric, coherence, by 1.4×. Notably, HARIN's mean reciprocal rank of 1 highlights its exceptional ability to recommend the optimal hierarchical topic model. We show that HARIN outperforms the coherence metric in identifying the best model while also pruning the right number of leaf topics, thereby enhancing both model selection and result interpretability.