  We explore a new class of brain encoding model by adding memory-related
information as input. Memory is an essential brain mechanism that works
alongside visual stimuli. During a vision-memory cognitive task, we found the
non-visual brain is largely predictable using previously seen images. Our
Memory Encoding Model (Mem) won the Algonauts 2023 visual brain competition
even without model ensemble (single model score 66.8, ensemble score 70.8). Our
ensemble model without memory input (61.4) can also stand a 3rd place.
Furthermore, we observe periodic delayed brain response correlated to 6th-7th
prior image, and hippocampus also showed correlated activity timed with this
periodicity. We conjuncture that the periodic replay could be related to memory
mechanism to enhance the working memory.
