Reconstructing Natural Scenes from fMRI Patters Using Deep Generative Networks

Milad MOZAFARI
Research Scientist at Torus AI
November 10th, 2022

Abstract
Decoding and reconstructing images from brain imaging data is a research area of high interest. Recent progress in deep generative neural networks has introduced new opportunities to tackle this problem. Here, we make use of large-scale generative networks to decode and reconstruct natural scenes from fMRI patterns. We computed a linear mapping between fMRI data, acquired over images from 150 different categories of ImageNet, and their corresponding deep network representations. Then, we applied this mapping to the fMRI activity patterns obtained from unseen test images in order to retrieve their latent vectors, and reconstruct the corresponding images. Not only was the pairwise image decoding from the predicted latent vectors highly accurate but also qualitative and quantitative assessments revealed that the resulting image reconstructions were visually plausible, successfully captured many attributes of the original images, and had high perceptual similarity with the original content.

Supplementary Materials
Paper 1: https://arxiv.org/pdf/2001.11761.pdf
Paper 2: https://arxiv.org/pdf/2202.12692.pdf