Document Type
Poster
Publication Title
Northrop Grumman Engineering & Science Student Design Showcase
Abstract
The use of an entropy-based loss function to improve BERT’s sentiment analysis on the Stanford Sentiment Treebank (SST-2) dataset. By studying entropy trends in a fine-tuned BERT model, we crafted a custom loss that stabilizes entropy in early layers (1–9) and penalizes entropy rises in later layers (10–12) using a mean entropy threshold. Our approach achieved 92.09% accuracy and a 92.31% F1 score, surpassing a cross-entropy baseline by 1.95%. These results highlight entropy-guided optimization’s potential for transformer models.
Advisor
Ryan White
Publication Date
4-25-2025
Recommended Citation
Batista de Moura, Pedro H., "Entropy-Guided Transformers for Sentiment Prediction" (2025). Mathematics and System Engineering Student Publications. 9.
https://repository.fit.edu/math_student/9