"Reflection in a line" maps any vector parallel to the line into itself. It maps any vector perpendicular to the line into its negative.
So: What is a vector in the direction of the line ax+ by? What is a vector in the direction of ax+ by= 0? What is a vector perpendicular to it?
Another way: find the rotation matrix, A, that rotates ax+ by= 0 to y= 0. The matrix that refleclects about y= 0 is . The matrix that reflects in the line ax+ by= 0 is .