Some complementary info to the other answer:
The original metaphors of these two words are different. This and their different original areas of application still has an effect on how and when they are used today.
Abonnieren is derived from the French verb abonner, which itself is derived from an Old French word for a boundary stone. The idea is that by paying a sum in advance you have buy the right to some periodical product up to a certain limit. It is only used for physical objects in the widest sense, including services that usually issue physical tickets (e.g. concerts). This excludes courses.
Though very rare today, in some cases the abonnement of a newspaper is implemented through a bunch of bons (coupons) which you can exchange for the real thing at your local news agent. I first thought that this was the origin of the verb, but apparently that's not the case. I still think it's relevant to the full set of connotations of the word, though.
Like English subscribe, abonnieren usually involves payment, though both words have been generalised to free subscriptions of, inter alia, email newsletter subscriptions.
The separable verb anmelden (register) is formed from melden (report, notify) by adding the preposition an (at). And in fact, registering something somewhere amounts to little more than reporting it at a place. One thing you can (an)melden is special card combinations in a card game. This has given rise to the technical term meld (used as a verb and as a noun) in English card game terminology for certain games, especially those with a German, Dutch or Yiddish connection.
The (near-)synonym of abonnieren is anmelden with a reflexive pronoun, i.e. sich anmelden. (I am not sure if this counts as a separate reflexive verb since its meaning seems to be entirely explained by the verb anmelden and the reflexive pronoun.) As the position of the direct object is already occupied by the reflexive pronoun, we have to supply the object of the subscription as an adverbial using a preposition such as für (for), bei (at) or zu (to). Usage is basically as in English: You register at a school for a course. Only where you register in English to get lessons, in German you register to the lessons themselves (without the verb get).
As is the case with register, anmelden connotes a certain amount of bureaucracy and commitment rather than a payment, about which it is agnostic.