  The human brain is a complex spiking neural network (SNN) that learns
multimodal signals in a zero-shot manner by generalizing existing knowledge.
Remarkably, the brain achieves this with minimal power consumption, using
event-based signals that propagate within its structure. However, mimicking the
human brain in neuromorphic hardware presents both hardware and software
challenges. Hardware limitations, such as the slowdown of Moore's law and the
von Neumann bottleneck, hinder the efficiency of digital computers. On the
software side, SNNs are known for their difficult training, especially when
learning multimodal signals. To overcome these challenges, we propose a
hardware-software co-design that combines a fixed and random liquid state
machine (LSM) SNN encoder with trainable artificial neural network (ANN)
projections. The LSM is physically implemented using analogue resistive memory,
leveraging the inherent stochasticity of resistive switching to generate random
weights. This highly efficient and nanoscale in-memory computing approach
effectively addresses the von Neumann bottleneck and the slowdown of Moore's
law. The ANN projections are implemented digitally, allowing for easy
optimization using contrastive loss, which helps to overcome the difficulties
associated with SNN training. We experimentally implement this co-design on a
40nm 256Kb in-memory computing macro. We first demonstrate LSM-based event
encoding through supervised classification and linear probing on the N-MNIST
and N-TIDIGITS datasets.
