I've been playing around with a few mods, and making custom icons for them, but the [item] links in the tooltips always have messed up small versions of the icons. I copied the mod.js example to my server, with the exact example icon sheet file, and it worked as expected. I literally opened the icon sheet file in photoshop and saved it as a new transparent png file, however, and I have the same result as the ones I created from scratch.
I don't see any significant differences between the two image files, both are 24-bit PNG images with transparency, so I'm missing something here.
For the curious, the problem was not anything inherent to the image file, but the fact that it wasn't hosted on the game server. A bug in the code changes any external links in the tooltip ("http://") to be unreadable.
I've included a fix in my modpack, until a permanent fix is released to the base game.