Fleece's Achilles' heel is that, without a nylon coating, it lacks any wind resistance--a slight breeze that wool and cotton would turn back cuts right through fleece with a nasty bite.
Lacking wind resistance is one such weakness of the fleece material.