I see a few potential gaps in your methods. One might be that you are using (presumably) constant axes for your
yAxis. I would guess those are the absolute world axes X and Y. That might be ok, if your specific scenario only uses planes oriented to X or Y. That's where you go awry.
To rotate from one direction to another, cross those angles
You cross one plane's normal and an absolute axis. That's not right.
vector3 axis = planeA.normal cross planeB.normal;
axis is only a vector with the same direction as the axis of rotation. A rotation axis needs to be a unit vector.
Cross products need to be normalized
The result of any cross-product,
bAxis or other, will only be a unit vector if both crossed vectors are unit vectors AND they are perpindicular to each other. This is a result of this property of cross products (and the fact that sin(90 degrees) = 1):
length(A x B) = sin(angle_between(A, B)) * length(A) * length(B)
// so if A and B are unit vectors (have length = 1)
length(A x B) = sin(angle_between(A, B))
// and if that length is 1...
1 = sin(angle_between(A, B)) = sin(90 degrees) // A and B were perpindicular
(Side note, you might be able to get a unit vector by crossing a weird combination of non-unit vectors and an angle that coincidentally still adds up to magnitude 1. But that's not the point, and both your planes' normal vectors ought to be unit vectors anyway.)
Rotation matrices are hard to build by hand
If you had normalized your axis, your matrix would still be very wrong. They are not constructed directly from the components of the participating vectors. This is a rotation matrix (again from wikipedia):
Moral of the story: use a library to construct your rotation matrix. You'll thank yourself later.
The correct axis and angle rotation matrix
Good news! You already have it after generating your cross product!
// length(A x B) = sin(angle_between(A, B)
// which leaves us with a handy:
vector3 a_cross_b = A x B;
float angle = arcsign(length(a_cross_b));
vector3 axis = normalize(a_cross_b);
matrix correct_rotation_matrix = create_rotation_from_axis_angle(axis, angle);
// a quaternion is also acceptable here.