As far as I know, "fall off" means to decline, or decrease the amount of something, and is intransitive.
The number of pandas in the world is falling off (decreasing) each year.
The membership fell off (decreased) dramatically when the chairperson resigned.
I understand what you mean, but, judging by the examples I've seen, I think I would say your sentence like this:
"As time went by, my nervosity fell off".
I'm sure some teacher can tell us something more.