'が' and 'か' are not determined by COLLATION utf8_unicode_ci, but by COLLATION utf8_general_ci.
The Japanese characters should be presented as '?' if you change your browser to use UTF-8.
Two characters in hexademical are '304b' and '304c' respectively.
The MySQL server I 'm using is 5.0.16-Debian_0.dotdeb.1-log on Debian.
sorry, I meant, the two characters should NOT be presented as '?' if you change your brower to use UTF-8.
and one row for x'304b'. So, they seems different in this version.
Of course they are different if you query with hexademicals.
My $LANG is en_AU.UTF-8 .
Please, try to repeat with a newer version, 5.0.19, and inform about the results. If the problem is still repeatable, please, attach a screenshots that demonstrates it.
Attachment: JPchar-bad-dertermination.JPG (image/jpeg, text), 40.54 KiB.
Attachment: working_collation.JPG (image/jpeg, text), 39.66 KiB.
These are the voiced/unvoiced components of the same letter pair, that's why they look almost the same.
in a case-insensitive collation for latin1.
Looking at the "Unicode Collation Algorithm" table 4.0.0, http://www.unicode.org/Public/UCA/4.0.0/allkeys-4.0.0.txt we find that HIRAGANA LETTER KA and HIRAGANA LETTER GA both have the same primary weight: 1E57. Following that, we must say that the characters are equal for searches.
You'll find the same thing for the next Hiragana letters in the chart, KI and GI (i.e. 304d = 304d), and so on.
We realize that people want different Japanese collation. They could get it with utf8_general_ci or with CONVERT(... to sjis), but admittedly those are usually bad solutions.
Plans exist for a new Japanese standard collation (work in progress).
So, it is not a bug, but a documented behaviour.
Well done Mr. Valeriy Kravchuk, but then I still have to go back to utf8_general_ci for now.