Of course he thought it was a joke and let it slide with a "hm?" but he was seriously annoyed. I mean, just don't say things like that even as a joke!
もちろん冗談と思って彼は「あっそ」と流していましたけどかなり嫌だったそうです。（ってゆうか冗談でもそんなこと言うな！）