Architectural bias in recurrent neural networks: Fractal analysis

Autor(en): Tino, P
Hammer, B
Stichwörter: Computer Science; Computer Science, Artificial Intelligence; DIMENSION; FINITE-STATE AUTOMATA; ITERATED FUNCTION SYSTEMS; LANGUAGES; Neurosciences; Neurosciences & Neurology; TIME
Erscheinungsdatum: 2003
Herausgeber: MIT PRESS
Volumen: 15
Ausgabe: 8
Startseite: 1931
Seitenende: 1957
We have recently shown that when initialized with ``small'' weights, recurrent neural networks (RNNs) with standard sigmoid-type activation functions are inherently biased toward Markov models; even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tino, 2002, Tino, Cernansky, & Benuskova, 2002a, 2002b). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this article, we extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finite-state transition diagram-a scenario that has been frequently considered in the past, for example, when studying RNN-based learning and implementation of regular grammars and finite-state transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as box counting and Hausdorff dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters, the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.
ISSN: 08997667
DOI: 10.1162/08997660360675099

Show full item record

Page view(s)

Last Week
Last month
checked on Feb 29, 2024

Google ScholarTM