Topological Mutual Information in Neural Language Models: An Information-Theoretic Approach to Semantic Structure Analysis
We propose a framework for analyzing semantic structure in neural language models through the lens of topological data analysis and information theory. Our approach computes persistent homology features from token embeddings and analyzes their correlations using mutual information, rather than treating semantic relationships as analogous to quantum systems. We develop enhanced attention mechanisms that incorporate topological mutual information to potentially capture hierarchical semantic relationships across multiple scales. While our contribution is primarily theoretical and methodological, we provide a complete implementation and demonstrate computational feasibility. The framework offers new tools for investigating whether semantic relationships exhibit detectable geometric structure and provides a foundation for future empirical research into the topological organizat
