Home >

news Help

Publication Information


Title
Japanese:Learning with Partial Forgetting in Modern Hopfield Networks 
English:Learning with Partial Forgetting in Modern Hopfield Networks 
Author
Japanese: Toshihiro Ota, Ikuro Sato, Rei Kawakami, Masayuki Tanaka, Nakamasa Inoue.  
English: Toshihiro Ota, Ikuro Sato, Rei Kawakami, Masayuki Tanaka, Nakamasa Inoue.  
Language English 
Journal/Book name
Japanese: 
English:Proceedings of The 26th International Conference on Artificial Intelligence and Statistics 
Volume, Number, Page        
Published date Apr. 2023 
Publisher
Japanese: 
English: 
Conference name
Japanese: 
English:The 26th International Conference on Artificial Intelligence and Statistics 
Conference site
Japanese: 
English:Valencia 
Official URL https://proceedings.mlr.press/v206/ota23a.html
 
Abstract It has been known by neuroscience studies that partial and transient forgetting of memory often plays an important role in the brain to improve performance for certain intellectual activities. In machine learning, associative memory models such as classical and modern Hopfield networks have been proposed to express memories as attractors in the feature space of a closed recurrent network. In this work, we propose learning with partial forgetting (LwPF), where a partial forgetting functionality is designed by element-wise non-bijective projections, for memory neurons in modern Hopfield networks to improve model performance. We incorporate LwPF into the attention mechanism also, whose process has been shown to be identical to the update rule of a certain modern Hopfield network, by modifying the corresponding Lagrangian. We evaluated the effectiveness of LwPF on three diverse tasks such as bit-pattern classification, immune repertoire classification for computational biology, and image classification for computer vision, and confirmed that LwPF consistently improves the performance of existing neural networks including DeepRC and vision transformers.

©2007 Tokyo Institute of Technology All rights reserved.