Yukiyaku Kamitani
2020-10-13 07:33:49
https://www.nature.com/articles/ncomms15037
Nature Communications
Machine learning algorithms can decode objects that people see or imagine from their brain activity. Here the authors present a predictive decoder combined with deep neural network representations that generalizes beyond the training set and correctly identifies novel objects that it has never been trained on.
Original URL: https://www.nature.com/articles/ncomms15037
Yukiyaku Kamitani
2020-10-13 07:34:29
https://www.frontiersin.org/articles/10.3389/fncom.2017.00004/full
Frontiers
Dreaming is generally thought to be generated by spontaneous brain activity during sleep with patterns common to waking experience. This view is supported by a recent study demonstrating that dreamed objects can be predicted from brain activity during sleep using statistical decoders trained with stimulus-induced brain activity. However, it remains unclear whether and how visual image features associated with dreamed objects are represented in the brain. In this study, we used a deep neural network (DNN) model for object recognition as a proxy for hierarchical visual feature representation, and DNN features for dreamed objects were analyzed with brain decoding of fMRI data collected during dreaming. The decoders were first trained with stimulus-induced brain activity labeled with the feature values of the stimulus image from multiple DNN layers. The decoders were then used to decode DNN features from the dream fMRI data, and the decoded features were compared with the averaged features of each object category calculated from a large-scale image database. We found that the feature values decoded from the dream fMRI data positively correlated with those associated with dreamed object categories at mid- to high-level DNN layers. Using the decoded features, the dreamed object category could be identified at above-chance levels by matching them to the averaged features for candidate categories. The results suggest that dreaming recruits hierarchical visual feature representatio...
Original URL: https://www.frontiersin.org/articles/10.3389/fncom.2017.00004/full
Yukiyaku Kamitani
2020-10-13 07:35:00
https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633
journals.plos.org
Author summary Machine learning-based analysis of human functional magnetic resonance imaging (fMRI) patterns has enabled the visualization of perceptual content. However, prior work visualizing perceptual contents from brain activity has failed to combine visual information of multiple hierarchical levels. Here, we present a method for visual image reconstruction from the brain that can reveal both seen and imagined contents by capitalizing on multiple levels of visual cortical representations. We decoded brain activity into hierarchical visual features of a deep neural network (DNN), and optimized an image to make its DNN features similar to the decoded features. Our method successfully produced perceptually similar images to viewed natural images and artificial images (colored shapes and letters), whereas the decoder was trained only on an independent set of natural images. It also generalized to the reconstruction of mental imagery of remembered images. Our approach allows for studying subjective contents represented in hierarchical neural representations by objectifying them into images.
Original URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006633
Yukiyaku Kamitani
2020-10-13 07:35:43
https://www.biorxiv.org/content/10.1101/2020.07.22.216713v1
bioRxiv
Achievement of human-level image recognition by deep neural networks (DNNs) has spurred interest in whether and how DNNs are brain-like. Both DNNs and the visual cortex perform hierarchical processing, and correspondence has been shown between hierarchical visual areas and DNN layers in representing visual features. Here, we propose the brain hierarchy (BH) score as a metric to quantify the degree of hierarchical correspondence based on the decoding of individual DNN unit activations from human brain activity. We find that BH scores for 29 pre-trained DNNs with varying architectures are negatively correlated with image recognition performance, indicating that recently developed high-performance DNNs are not necessarily brain-like. Experimental manipulations of DNN models suggest that relatively simple feedforward architecture with broad spatial integration is critical to brain-like hierarchy. Our method provides new ways for designing DNNs and understanding the brain in consideration of their representational homology. ### Competing Interest Statement The authors have declared no competing interest.
Original URL: https://www.biorxiv.org/content/10.1101/2020.07.22.216713v1
Yukiyaku Kamitani
2020-10-13 07:45:36
https://www.youtube.com/watch?v=jsp1KaM-avU&feature=youtu.be
YouTube
Original URL: https://www.youtube.com/watch?v=jsp1KaM-avU&feature=youtu.be