A more compelling explanation is that Ellison wanted to write a second novel that would meet the standard of Invisible Man while being an entirely different kind of book.
Invisible Man is a novel that Ellison knew about.