<blockquote>
  <p>So entropy seeds mnemonic (bip39?), then mnemonic | password hashes to Seed; Seed then acts as the master key for the bip32 xprv? (correct me if I'm wrong!?)</p>
</blockquote>

<p>That sounds about right. Most of the process is well detailed in <a href="https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki" rel="nofollow noreferrer">BIP-39</a>.</p>

<ol>
<li>An SHA-256 is taken of the entropy, and the first <code>entropy_len_in_bits / 32</code> bits of this hash are appended to the end of the entropy. The resulting entropy bit string is divisible into 11-bit-long chunks (it's no longer an integral number of bytes).</li>
<li>Each 11-bit chunk is converted into one of 2<sup>11</sup> mnemonic words.</li>
<li>The words are joined by spaces. For display purposes in Japanese, these should be Unicode IDEOGRAPHIC SPACEs, <code>'\u3000'</code>. If there's no need to display the mnemonic to the user, they can be "normal" SPACEs (<code>'\u0020'</code>).</li>
<li>The mnemonic sentence is <a href="http://en.wikipedia.org/wiki/Unicode_equivalence#Normalization" rel="nofollow noreferrer">Unicode normalized</a> in NFKD form. This converts any IDEOGRAPHIC SPACEs into SPACEs. It also changes some characters in some of the mnemonic words, therefore this step cannot be skipped. (The question <em>What is NFKD normalization?</em> is a whole separate topic that's probably best asked elsewhere IMO....)</li>
<li>The mnemonic sentence is converted into bytes via <a href="http://en.wikipedia.org/wiki/UTF-8" rel="nofollow noreferrer">UTF-8 encoding</a>.</li>
<li>The binary seed is computed as PBKDF2<sub>HMAC<sub>SHA512</sub></sub>(key= "mnemonic" | passphrase, data=utf8_mnemonic, iterations=2048, out_bytes_length=64). The passphrase can be the empty string. It must first go through the same steps 4 and 5 as the mnemonic.</li>
<li>(this part isn't detailed anywhere AFAIK) The master extended private key is constructed by using the first 32 bytes of the binary seed as the private key, and the last 32 bytes as the chaincode.</li>
</ol>

<blockquote>
  <p>Is it just NKFD Unicode normalization, which Electrum 2.0 does?</p>
</blockquote>

<p>Electrum 2.x does use NFKD normalization, but it also performs additional steps, such as removing spaces between Japanese words after step 4. It also uses a different key string in step 6, and a completely different process prior to step 4. See <a href="https://bitcoin.stackexchange.com/questions/37176/how-does-electrum-make-a-keypair-out-of-a-seed#37180">this answer</a> for an implementation of Electrum 2.x's mnemonic-words-to-seed procedure in Python.</p>

CC BY-SA 3.0
2015-06-04T11:18:33.150Z
Thanks for the response. I've got gaps in how UTF-8 fits in, though I understand pretty well what NKFD does. **Why not just encode Unicode directly?** Also, I do understand the ideographic space, I think (\u3000 is the Japanese "equivalent" of \u0020). It was actually [Electrum 2.0's implementation of the seed preparation](https://github.com/simcity4242/pybitcointools/blob/master/bitcoin/deterministic.py%23L201) which was confusing me. That really is strange Electrum deviated in such obscure places
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-04T11:22:01.300Z
Scratch that. You can't encode Unicode 4 byte code points as well as UTF8, right?
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-04T11:27:29.953Z
@WizardOfOzzie Unicode strings are simply sequences of integers in the range [0,0x10FFFF]. Before you can hash it, you need to convert it into a sequence of bytes. A simple way is to just take each 4-byte int, and use those bytes as-is (UTF-32LE encoding), but this is inneficient from a space point of view (considering that English only needs 1 byte per character). UTF-8 is more complicated, but more space efficient most of the time.
Christopher Gurnee
CC BY-SA 3.0
2015-06-04T11:34:16.163Z
@WizardOfOzzie Agreed that Electrum 2.x's normalization is much more complex, but it's helpful to minimize the chance of loss from mistyped mnemonics given the lack of any specific wordlists. (And BIP-39's requirement for specific wordlists was something Electrum 2.x's dev really disagreed with.)
Christopher Gurnee
CC BY-SA 3.0
2015-06-04T22:26:15.627Z
Does this look right? `norm = lambda d: (' '.join(unicodedata.('NFKD', unicode(d)).split('\u3000'))).encode('utf-8')`
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-04T22:47:31.220Z
Assuming `d` is a mnemonic sentence of Python2 type string or a unicode? I'd think something like this for most languages (different for Chinese, which might not have spaces in `d`): `norm = lambda d: (u' '.join(unicodedata.normalize('NFKD', unicode(d)).split())).encode('utf-8')` (I added the missing "normalize" and changed split to split on all whitespace). Note that if you're accepting input from a user, BIP-39 requires that you verify the checksum.
Christopher Gurnee
CC BY-SA 3.0
2015-06-04T23:25:20.207Z
Thx, I'll try it now. Hmmm, assuming py 2/3 so let's say both Unicode and str. Yep, I'm pretty well versed with bip39 so my mn2hex function asserts check_bip39
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-09T04:35:23.577Z
I've gotten all the unit tests to work *except for the comparison of `bip39_hex_to_mn` with `VECTOR['mnemonic']` as the function returns a standard space (ie `\u0020`) whereas the test vectors use `\u3000`. Electrum (2.x) solves the ideographic space (`\u3000`) problem by concatenating all CJK words without spaces. See https://github.com/simcity4242/pybitcointools/blob/master/test_bip39.py#L123 (I've only got it to work with `u' '.join(v['mnemonic'].split())`, whereas it should just be `v['mnemonic']`
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-09T04:42:25.387Z
Tangentially related, why do the word lists for English and Japanese have the correct format (each word is a unicode object consisting of unicode characters), whereas the Spanish, Chinese and French (https://github.com/simcity4242/pybitcointools/blob/master/bitcoin/_bip39.py#L2349 and below) wordlists are of this format: `'\xe7\x9a\x84'`?
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-09T14:04:26.803Z
@WizardOfOzzie By my reading of the standard, bip39_hex_to_mn() should return mnemonics with ideographic spaces for Japanese, so that the result can be used for both display purposes and for calculating the binary seed. As it's currently written, the returned value is only suitable for the latter purpose. Given this change, you could remove the "hack" you made to TestBIP39JAP.
Christopher Gurnee
CC BY-SA 3.0
2015-06-09T14:22:39.610Z
@WizardOfOzzie Regarding your word lists, I agree with you that I'd prefer they all be unicode objects for consistency. Where did you get your word lists? Perhaps you should download the official BIP-39 word list text files directly from [the repo](https://github.com/bitcoin/bips/tree/master/bip-0039), and load them with something like `with io.open(language+'.txt', encoding='utf_8_sig') as words_file: words[language] = tuple(word.strip() for word in words_file)`.
Christopher Gurnee
CC BY-SA 3.0
2015-06-10T03:24:10.353Z
I loaded them using iPython, then wrote the variable to a file with `%store var >> file.py`. I'll update both recommendations, thanks!
Wizard Of Ozzie
CC BY-SA 3.0
2015-06-04T11:09:58.400Z
2015-06-04T11:09:58.400Z
2017-04-13T12:47:33.610Z
Christopher Gurnee
<p>There is a <a href="https://github.com/bip32JP/bip32JP.github.io/blob/master/test_JP_BIP39.json" rel="nofollow">json for unit tests of Japanese characters</a> which I want to validate using Python, specifically with <a href="https://github.com/simcity4242/pybitcointools" rel="nofollow">this fork of pybitcointools</a>, which has bip39 functionality.</p>

<p>Unit tests from <a href="https://github.com/simcity4242/pybitcointools/blob/master/test_bip39.py" rel="nofollow">Trezor's python-mnemonic test vectors</a> work fine (in Python 2.7 IME), however, this is straightforward since there's no normalization of unicode dialectics and such, since all mnemonics are lower case English. </p>

<p>The Japanese fields are:</p>

<ol>
<li>Entropy (hex)</li>
<li>Mnemonic (Japanese)</li>
<li>Password (Japanese, appears to be the same for all tests)</li>
<li>Seed (hex, 64 bytes)</li>
<li><code>xprv</code></li>
</ol>

<p>So <code>entropy</code> seeds <code>mnemonic (bip39?)</code>, then <code>mnemonic</code> | <code>password</code> hashes to <code>Seed</code>; <code>Seed</code> then acts as the master key for the bip32 xprv? (correct me if I'm wrong!?)</p>

<p>So, assuming it's that straightforward... </p>

<ol>
<li><em>how is the Japanese unicode text "normalized"?</em> (Is it just NKFD Unicode normalization, which <a href="https://github.com/spesmilo/electrum/blob/master/lib/mnemonic.py%23L34" rel="nofollow">Electrum 2.0</a> does?)</li>
<li><em>what does "normal" mean for Japanese?</em></li>
</ol>

<development><python><testing><mnemonic-seed>
BIP39 Japanese Mnemonic vector unit test process
Wizard Of Ozzie
# BIP39 Japanese Mnemonic vector unit test process

(Asked by: username_0 on 2015-06-04)

There is a [json for unit tests of Japanese characters](https://github.com/bip32JP/bip32JP.github.io/blob/master/test_JP_BIP39.json) which I want to validate using Python, specifically with [this fork of pybitcointools](https://github.com/simcity4242/pybitcointools), which has bip39 functionality.

Unit tests from [Trezor's python-mnemonic test vectors](https://github.com/simcity4242/pybitcointools/blob/master/test_bip39.py) work fine (in Python 2.7 IME), however, this is straightforward since there's no normalization of unicode dialectics and such, since all mnemonics are lower case English.

The Japanese fields are:

1. Entropy (hex)
2. Mnemonic (Japanese)
3. Password (Japanese, appears to be the same for all tests)
4. Seed (hex, 64 bytes)
5. `xprv`

So `entropy` seeds `mnemonic (bip39?)`, then `mnemonic` | `password` hashes to `Seed`; `Seed` then acts as the master key for the bip32 xprv? (correct me if I'm wrong!?)

So, assuming it's that straightforward...

1. *how is the Japanese unicode text "normalized"?* (Is it just NKFD Unicode normalization, which [Electrum 2.0](https://github.com/spesmilo/electrum/blob/master/lib/mnemonic.py%23L34) does?)
2. *what does "normal" mean for Japanese?*

**Tags:** development, python, testing, mnemonic-seed


---

## 1 answers

### Answer by username_1 on 2015-06-04. Score: 2 | Accepted answer

> So entropy seeds mnemonic (bip39?), then mnemonic | password hashes to Seed; Seed then acts as the master key for the bip32 xprv? (correct me if I'm wrong!?)

That sounds about right. Most of the process is well detailed in [BIP-39](https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki).

1. An SHA-256 is taken of the entropy, and the first `entropy_len_in_bits / 32` bits of this hash are appended to the end of the entropy. The resulting entropy bit string is divisible into 11-bit-long chunks (it's no longer an integral number of bytes).
2. Each 11-bit chunk is converted into one of 211 mnemonic words.
3. The words are joined by spaces. For display purposes in Japanese, these should be Unicode IDEOGRAPHIC SPACEs, `'\u3000'`. If there's no need to display the mnemonic to the user, they can be "normal" SPACEs (`'\u0020'`).
4. The mnemonic sentence is [Unicode normalized](http://en.wikipedia.org/wiki/Unicode_equivalence#Normalization) in NFKD form. This converts any IDEOGRAPHIC SPACEs into SPACEs. It also changes some characters in some of the mnemonic words, therefore this step cannot be skipped. (The question *What is NFKD normalization?* is a whole separate topic that's probably best asked elsewhere IMO....)
5. The mnemonic sentence is converted into bytes via [UTF-8 encoding](http://en.wikipedia.org/wiki/UTF-8).
6. The binary seed is computed as PBKDF2HMACSHA512(key= "mnemonic" | passphrase, data=utf8\_mnemonic, iterations=2048, out\_bytes\_length=64). The passphrase can be the empty string. It must first go through the same steps 4 and 5 as the mnemonic.
7. (this part isn't detailed anywhere AFAIK) The master extended private key is constructed by using the first 32 bytes of the binary seed as the private key, and the last 32 bytes as the chaincode.

> Is it just NKFD Unicode normalization, which Electrum 2.0 does?

Electrum 2.x does use NFKD normalization, but it also performs additional steps, such as removing spaces between Japanese words after step 4. It also uses a different key string in step 6, and a completely different process prior to step 4. See [this answer](https://bitcoin.stackexchange.com/questions/37176/how-does-electrum-make-a-keypair-out-of-a-seed#37180) for an implementation of Electrum 2.x's mnemonic-words-to-seed procedure in Python.

> **Comments:**
> **username_0**: Thanks for the response. I've got gaps in how UTF-8 fits in, though I understand pretty well what NKFD does. **Why not just encode Unicode directly?** Also, I do understand the ideographic space, I think (\u3000 is the Japanese "equivalent" of \u0020). It was actually [Electrum 2.0's implementation of the seed preparation](https://github.com/simcity4242/pybitcointools/blob/master/bitcoin/deterministic.py%23L201) which was confusing me. That really is strange Electrum deviated in such obscure places  
> **username_0**: Scratch that. You can't encode Unicode 4 byte code points as well as UTF8, right?  
> **username_1**: @username_0 Unicode strings are simply sequences of integers in the range [0,0x10FFFF]. Before you can hash it, you need to convert it into a sequence of bytes. A simple way is to just take each 4-byte int, and use those bytes as-is (UTF-32LE encoding), but this is inneficient from a space point of view (considering that English only needs 1 byte per character). UTF-8 is more complicated, but more space efficient most of the time.  
> **username_1**: @username_0 Agreed that Electrum 2.x's normalization is much more complex, but it's helpful to minimize the chance of loss from mistyped mnemonics given the lack of any specific wordlists. (And BIP-39's requirement for specific wordlists was something Electrum 2.x's dev really disagreed with.)  
> **username_0**: Does this look right? `norm = lambda d: (' '.join(unicodedata.('NFKD', unicode(d)).split('\u3000'))).encode('utf-8')`  
> **username_1**: Assuming `d` is a mnemonic sentence of Python2 type string or a unicode? I'd think something like this for most languages (different for Chinese, which might not have spaces in `d`): `norm = lambda d: (u' '.join(unicodedata.normalize('NFKD', unicode(d)).split())).encode('utf-8')` (I added the missing "normalize" and changed split to split on all whitespace). Note that if you're accepting input from a user, BIP-39 requires that you verify the checksum.  
> **username_0**: Thx, I'll try it now. Hmmm, assuming py 2/3 so let's say both Unicode and str. Yep, I'm pretty well versed with bip39 so my mn2hex function asserts check_bip39  
> **username_0**: I've gotten all the unit tests to work *except for the comparison of `bip39_hex_to_mn` with `VECTOR['mnemonic']` as the function returns a standard space (ie `\u0020`) whereas the test vectors use `\u3000`. Electrum (2.x) solves the ideographic space (`\u3000`) problem by concatenating all CJK words without spaces. See https://github.com/simcity4242/pybitcointools/blob/master/test_bip39.py#L123 (I've only got it to work with `u' '.join(v['mnemonic'].split())`, whereas it should just be `v['mnemonic']`  
> **username_0**: Tangentially related, why do the word lists for English and Japanese have the correct format (each word is a unicode object consisting of unicode characters), whereas the Spanish, Chinese and French (https://github.com/simcity4242/pybitcointools/blob/master/bitcoin/_bip39.py#L2349 and below) wordlists are of this format: `'\xe7\x9a\x84'`?  
> **username_1**: @username_0 By my reading of the standard, bip39_hex_to_mn() should return mnemonics with ideographic spaces for Japanese, so that the result can be used for both display purposes and for calculating the binary seed. As it's currently written, the returned value is only suitable for the latter purpose. Given this change, you could remove the "hack" you made to TestBIP39JAP.  
> **username_1**: @username_0 Regarding your word lists, I agree with you that I'd prefer they all be unicode objects for consistency. Where did you get your word lists? Perhaps you should download the official BIP-39 word list text files directly from [the repo](https://github.com/bitcoin/bips/tree/master/bip-0039), and load them with something like `with io.open(language+'.txt', encoding='utf_8_sig') as words_file: words[language] = tuple(word.strip() for word in words_file)`.  
> **username_0**: I loaded them using iPython, then wrote the variable to a file with `%store var >> file.py`. I'll update both recommendations, thanks!  

---

