  The continued evolution and diversity of malware constitutes a major threat
in modern systems. It is well proven that security defenses currently available
are ineffective to mitigate the skills and imagination of cyber-criminals
necessitating the development of novel solutions. Deep learning algorithms and
artificial intelligence (AI) are rapidly evolving with remarkable results in
many application areas. Following the advances of AI and recognizing the need
for efficient malware detection methods, this paper presents a new approach for
malware detection based on binary visualization and self-organizing incremental
neural networks. The proposed method's performance in detecting malicious
payloads in various file types was investigated and the experimental results
showed that a detection accuracy of 91.7% and 94.1% was achieved for ransomware
in .pdf and .doc files respectively. With respect to other formats of malicious
code and other file types, including binaries, the proposed method behaved well
with an incremental detection rate that allows efficiently detecting unknown
malware at real-time.
