Opening theBlack Box: Analyzing Attention Weights andHidden States inPre-trained Language Models forNon-language Tasks
Autor(en): | Ballout, Mohamad Krumnack, Ulf Heidemann, Gunther Kühnberger, Kai-Uwe |
Herausgeber: | Longo, L. | Stichwörter: | Attention analyse; Attention analysis; BERT; Black boxes; Computational linguistics; Deep learning; Hidden state; Language model; Learning languages; Pre-trained language model; Research areas; Statistical tests; Transformer; Transformers; XAI | Erscheinungsdatum: | 2023 | Herausgeber: | Springer Science and Business Media Deutschland GmbH | Enthalten in: | Communications in Computer and Information Science | Band: | 1903 CCIS | Startseite: | 3 – 25 | Zusammenfassung: | Investigating deep learning language models has always been a significant research area due to the “black box” nature of most advanced models. With the recent advancements in pre-trained language models based on transformers and their increasing integration into daily life, addressing this issue has become more pressing. In order to achieve an explainable AI model, it is essential to comprehend the procedural steps involved and compare them with human thought processes. Thus, in this paper, we use simple, well-understood non-language tasks to explore these models' inner workings. Specifically, we apply a pre-trained language model to constraithmetic problems with hierarchical structure, to analyze their attention weight scores and hidden states. The investigation reveals promising results, with the model addressing hierarchical problems in a moderately structured manner, similar to human problem-solving strategies. Additionally, by inspecting the attention weights layer by layer, we uncover an unconventional finding that layer 10, rather than the model's final layer, is the optimal layer to unfreeze for the least parameter-intensive approach to fine-tune the model. We support these findings with entropy analysis and token embeddings similarity analysis. The attention analysis allows us to hypothesize that the model can generalize to longer sequences in ListOps dataset, a conclusion later confirmed through testing on sequences longer than those in the training set. Lastly, by utilizing a straightforward task in which the model predicts the winner of a Tic Tac Toe game, we identify limitations in attention analysis, particularly its inability to capture 2D patterns. © 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG. |
Beschreibung: | Cited by: 0; Conference name: 1st World Conference on eXplainable Artificial Intelligence, xAI 2023; Conference date: 26 July 2023 through 28 July 2023; Conference code: 303319; All Open Access, Green Open Access |
ISBN: | 9783031440694 | ISSN: | 1865-0929 | DOI: | 10.1007/978-3-031-44070-0_1 | Externe URL: | https://www.scopus.com/inward/record.uri?eid=2-s2.0-85176017702&doi=10.1007%2f978-3-031-44070-0_1&partnerID=40&md5=68ebad5f73324dadc56659fa9c225a45 |
Zur Langanzeige
Seitenaufrufe
3
Letzte Woche
0
0
Letzter Monat
0
0
geprüft am 08.06.2024