<p>As long as the underlying OS image is susceptible to changes, you will not get a reproducible build system that guarantees reproducibility over a long period of time. If the OS can update compiler toolchains, libc versions and whatever tools are used during a gitian build, changes in the future to the output are expected. Since this is the case since gitian is ubuntu based, and patches are still shipped to older ubuntu versions, this is to be expected after some time.</p>

CC BY-SA 4.0
2020-11-12T22:21:54.600Z
2020-11-12T22:21:54.600Z
TheCharlatan
<p>I've done a rebuild of 0.20.1 and I get the same results that you do. This would indicate that a build dependency has updated to produce slightly different results than the version that was in use at the time of the release. The build dependency versions are pinned unlike the actual software dependencies. IIRC this is common for gitian builds and attempting to rebuild even older releases will result in similar mismatches.</p>
<p>There is ongoing work to move to a different reproducible builds system called guix. This build system will pin the exact dependency versions, along with the option to build the entire toolchain from scratch. This should enable fully reproducible builds that can be replicated at any point in time.</p>
<hr />
<p>Here is a more in depth explanation of this particular mismatch, copied from my response <a href="https://github.com/bitcoin/bitcoin/issues/20389#issuecomment-727527843" rel="nofollow noreferrer">in this GitHub issue</a>.</p>
<p>If you look at the gitian build results, you will see a file named <code>bitcoin-0.20.1-x86_64-linux-gnu-debug.tar.gz</code>. If you untar this file, there will be <code>*.dbg</code> files, e.g. <code>bitcoind.dbg</code> and <code>bitcoin-qt.dbg</code>. These <code>*.dbg</code> files contain the debug data for their respective binaries, i.e. <code>bitcoind.dbg</code> contains debug data for <code>bitcoind</code>.</p>
<p>To ensure that you use the correct <code>dbg</code> file with the correct binary, gcc embeds a checksum of the <code>dbg</code> file within the binary itself. This means that <code>bitcoind</code> contains a checksum of <code>bitcoind.dbg</code>. This is to prevent attempting to debug <code>bitcoind</code> with another <code>dbg</code> file. For example, if you attempted to tell gdb that the <code>bitcoin-qt.dbg</code> file contained the debugging symbols for <code>bitcoind</code>, it would detect it does not and not attempt to load debugging symbols from <code>bitcoin-qt.dbg</code>.</p>
<p>The debugging symbols are initially compiled into the <code>bitcoind</code> binary, but later during the gitian build these are removed and placed into the <code>bitcoind.dbg</code> file. However because they are compiled into <code>bitcoind</code> initially, the debugging symbols have an effect on the build id that gcc embeds into the binary. The build ID is a hash of the compiled binary, including the debugging data.</p>
<p>So the end result is that the published binary contains two commitments to the debug symbols, but does not actually contain them itself.</p>
<p>You can read more about these separate debug files here: <a href="https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html" rel="nofollow noreferrer">https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html</a></p>
<p>The crux of this issue is that the debug symbols from recent gitian builds differ from the debug symbols that were generated for the original release. This means that both the build ID and the debug symbol checksum that we find inside of <code>bitcoind</code> are different. This then results in the <code>bitcoind</code> hashes being different (as well as all of the other binaries). And of course that causes the tarfile hashes to be different.</p>
<hr />
<p>Here is the diff of the binaries that diffoscope generates:</p>
<pre><code>--- bitcoind
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│  
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│  
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│  
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ @@ -1,5 +1,5 @@
│  
│  Hex dump of section '.gnu_debuglink':
│    0x00000000 62697463 6f696e64 2e646267 00000000 bitcoind.dbg....
│ -  0x00000010 b25ceebb                            .\..
│ +  0x00000010 114d519d                            .MQ.
</code></pre>
<p>As you can see, there are only two differences here, one in the build ID, and one in the <code>.gnu_debuglink</code> section. From the documentation I linked earlier, we can see that this <code>.gnu_debuglink</code> section has the first line is the debug filename followed by enough 0 bytes to pad to a 4 byte boundary. The second line is the 4 byte CRC checksum. And it is this CRC checksum that differs.</p>
<p>So why do the debug symbols differ here? Again, diffoscope can help us a bit.</p>
<pre><code>--- bitcoind.dbg
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind.dbg
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│  
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│  
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│  
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --debug-dump=info {}
│┄ error from `readelf --wide --debug-dump=info {}`:
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│ @@ -85052,36 +85052,36 @@
│      &lt;29607&gt;   DW_AT_decl_line   : 124
│      &lt;29608&gt;   DW_AT_decl_column : 16
│      &lt;29609&gt;   DW_AT_type        : &lt;0x28761&gt;
│      &lt;2960d&gt;   DW_AT_data_member_location: 12
│   &lt;2&gt;&lt;2960e&gt;: Abbrev Number: 30 (DW_TAG_member)
│      &lt;2960f&gt;   DW_AT_name        : (indirect string, offset: 0x13aaf): __kind
│      &lt;29613&gt;   DW_AT_decl_file   : 108
│ -    &lt;29614&gt;   DW_AT_decl_line   : 148
│ +    &lt;29614&gt;   DW_AT_decl_line   : 128
│      &lt;29615&gt;   DW_AT_decl_column : 7
│      &lt;29616&gt;   DW_AT_type        : &lt;0x287a6&gt;
│      &lt;2961a&gt;   DW_AT_data_member_location: 16
│   &lt;2&gt;&lt;2961b&gt;: Abbrev Number: 30 (DW_TAG_member)
│      &lt;2961c&gt;   DW_AT_name        : (indirect string, offset: 0x7a535): __spins
│      &lt;29620&gt;   DW_AT_decl_file   : 108
│ -    &lt;29621&gt;   DW_AT_decl_line   : 154
│ +    &lt;29621&gt;   DW_AT_decl_line   : 134
│      &lt;29622&gt;   DW_AT_decl_column : 3
│      &lt;29623&gt;   DW_AT_type        : &lt;0x2879a&gt;
│      &lt;29627&gt;   DW_AT_data_member_location: 20
...
</code></pre>
<p>There's a lot more output that I haven't included because it's pretty much all of the same.</p>
<p>Now this isn't terribly helpful, but we can see that for a bunch of functions, the line number for that function differs by 20 lines.</p>
<p>To get some more information, I used dwarfdump. This is what it says for the new build for the two functions I show in diffoscope (the functions are __kind and __spins).</p>
<pre><code>0x0002960e:     DW_TAG_member
                  DW_AT_name    (&quot;__kind&quot;)
                  DW_AT_decl_file       (&quot;/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h&quot;)
                  DW_AT_decl_line       (148)
                  DW_AT_decl_column     (0x07)
                  DW_AT_type    (0x000287a6 &quot;int&quot;)
                  DW_AT_data_member_location    (0x10)

0x0002961b:     DW_TAG_member
                  DW_AT_name    (&quot;__spins&quot;)
                  DW_AT_decl_file       (&quot;/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h&quot;)
                  DW_AT_decl_line       (154)
                  DW_AT_decl_column     (0x03)
                  DW_AT_type    (0x0002879a &quot;short int&quot;)
                  DW_AT_data_member_location    (0x14)
</code></pre>
<p>As you can see by the given file name, these functions come from libraries installed to the system. These appear to be headers for gcc's implementation of the c++ stdlib.</p>
<hr />
<p>So what's happened is that libstdc++ has updated in Ubuntu. Whatever updates hapened have moved some code in some header files that Bitcoin Core includes in its use of the c++ stdlib. In turn, compiling with those updated headers results in different debug symbols because function declarations have moved in those header files. This then results in gcc computing a different build ID and a different CRC checksum for the debug symbols. This lastly results in the final binaries being slightly different, which causes the hashes to mismatch.</p>

CC BY-SA 4.0
2020-11-15T10:37:50.277Z
I'm curious as to the choice of guix over Nix - do you have a link to any discussion on irc, github, etc. on why guix was chosen?
Raghav Sood
CC BY-SA 4.0
2020-11-15T17:36:11.973Z
@RaghavSood From https://github.com/bitcoin/bitcoin/pull/15277: "Guix is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on bootstrappability and reproducibility which are attractive for security-sensitive projects like bitcoin." I don't think there has been much discussion about guix vs nix, just that guix is what someone has put the effort into doing, so it's what we get.
Ava Chow
CC BY-SA 4.0
2020-11-15T22:44:08.503Z
Thanks for the detailed info. If it's interesting I have cmp output here comparing both bitcoind files. 
https://github.com/devrandom/gitian-builder/issues/235
Luke-jr let me know the files it looked like were in question were:
.note.gnu.build-id .gnu_debuglink

@AndrewChow you think this is a design flaw or working as intended?
Michael Tidwell
CC BY-SA 4.0
2020-11-15T23:10:18.520Z
It is a design flaw of the gitian system, but the gitian system is also working as intended.
Ava Chow
CC BY-SA 4.0
2020-11-12T03:39:40.340Z
2020-11-15T07:31:41.917Z
2020-11-15T07:31:41.917Z
Ava Chow
<p>Bitcoincore.org and Bitcoin.org will let you download a tarball for bitcoin v0.20.1</p>
<h3>When they do it</h3>
<p><strong>bitcoind:</strong>
<code>4ec74161b2a90293926ae8e20a2efbe952bd23b53aeebf051e6a6285ace18271</code><br>
<strong>bitcoin-0.20.1-x86_64-linux-gnu.tar.gz:</strong>
<code>376194f06596ecfa40331167c39bc70c355f960280bd2a645fdbf18f66527397</code></p>
<p>They also say:</p>
<blockquote>
<p>reproducing a binary for yourself will provide you with the highest
level of assurance currently available</p>
</blockquote>
<p>So, it seems like the current method to do this is the &quot;gitian build&quot; maneuver.</p>
<h3>When I do it</h3>
<p><strong>bitcoind:</strong>
<code>c5fb850ed9e6afa0af5653e743084912c9bd71fb233acfe2d72738fc319f2181</code><br>
<strong>bitcoin-0.20.1-x86_64-linux-gnu.tar.gz:</strong>
<code>277599356bd2df760832c6636797fe5ea5a5c28d929d53635b685f5ac1e4689b</code></p>
<p>Went through the gitian process with <strong>Ubuntu 20.04</strong> and <strong>Debian 10</strong>.
Both produced the same incorrect? tarball, 2 different setups/ <strong>2 Operating Systems, same tarball.</strong></p>
<p>Both my assert files are uploaded here for Debian and Ubuntu.
<a href="https://github.com/miketwenty1/images/tree/master/bad_asserts" rel="nofollow noreferrer">https://github.com/miketwenty1/images/tree/master/bad_asserts</a></p>
<hr />
<p>As a pleb I can think 1 of 2 things..</p>
<p>Something is wrong/off with some of my packages / versions / env / OS / setup OR, All the verifiers are lying and bitcoin's been compromised. Help me narrow down the possibilities.</p>
<p>I would like to go through this exercise properly and reproduce the same binaries.. but unsure of the best way forward.. what are the next steps to verifying the build for v0.20.1? I can't seem to reproduce the tarball advertised on bitcoincore . org and bitcoin . org. I can however make a reproducible build which makes me question a lot of things.</p>
<p>Would love if someone could go through their process from scratch and see if they can produce the same tarballs seen in this repo by these people. <a href="https://github.com/bitcoin-core/gitian.sigs/blob/master/0.20.1-linux" rel="nofollow noreferrer">https://github.com/bitcoin-core/gitian.sigs/blob/master/0.20.1-linux</a> /*/ <code>bitcoin-core-linux-0.20-build.assert</code>'s.
And share their complete process with me. After reproducing myself I would be happy to make a current walkthrough on the method and update the Docs, which (links below) are dated and broken for nearly all methods.</p>
<p>NOTE: If this is indeed the defacto way to currently do reproducible builds.. it would be nice to have documentation that is precise and current. I understand Ginx is something being worked on.. but would be nice to have a current doc in the interim for people who wants to verify builds. I would also be very satisfied if someone pointed something I was missing.</p>
<p><strong>reference links:</strong><br>
<a href="https://gitian.org/" rel="nofollow noreferrer">https://gitian.org/</a><br>
<a href="https://github.com/bitcoin-core/docs/blob/master/gitian-building.md" rel="nofollow noreferrer">https://github.com/bitcoin-core/docs/blob/master/gitian-building.md</a><br>
<a href="https://github.com/bitcoin-core/docs/tree/master/gitian-building" rel="nofollow noreferrer">https://github.com/bitcoin-core/docs/tree/master/gitian-building</a><br>
<a href="https://github.com/devrandom/gitian-builder/blob/master/README.md" rel="nofollow noreferrer">https://github.com/devrandom/gitian-builder/blob/master/README.md</a><br>
<a href="https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.py" rel="nofollow noreferrer">https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.py</a></p>
<p><strong>reference github issue</strong><br>
<a href="https://github.com/devrandom/gitian-builder/issues/235" rel="nofollow noreferrer">https://github.com/devrandom/gitian-builder/issues/235</a></p>

CC BY-SA 4.0
2020-11-14T16:42:24.533Z
FYI, reported the problem at https://github.com/bitcoin/bitcoin/issues/20389
Martin Monperrus
<bitcoin-core><bitcoind><compiling><gitian><guix>
Reproducible Gitian Builds .. but not the same hash as bitcoincore.org
Michael Tidwell
# Reproducible Gitian Builds .. but not the same hash as bitcoincore.org

(Asked by: username_0 on 2020-11-12)

Bitcoincore.org and Bitcoin.org will let you download a tarball for bitcoin v0.20.1

### When they do it

**bitcoind:**
`4ec74161b2a90293926ae8e20a2efbe952bd23b53aeebf051e6a6285ace18271`  
**bitcoin-0.20.1-x86\_64-linux-gnu.tar.gz:**
`376194f06596ecfa40331167c39bc70c355f960280bd2a645fdbf18f66527397`

They also say:

> reproducing a binary for yourself will provide you with the highest
> level of assurance currently available

So, it seems like the current method to do this is the "gitian build" maneuver.

### When I do it

**bitcoind:**
`c5fb850ed9e6afa0af5653e743084912c9bd71fb233acfe2d72738fc319f2181`  
**bitcoin-0.20.1-x86\_64-linux-gnu.tar.gz:**
`277599356bd2df760832c6636797fe5ea5a5c28d929d53635b685f5ac1e4689b`

Went through the gitian process with **Ubuntu 20.04** and **Debian 10**.
Both produced the same incorrect? tarball, 2 different setups/ **2 Operating Systems, same tarball.**

Both my assert files are uploaded here for Debian and Ubuntu.
<https://github.com/miketwenty1/images/tree/master/bad_asserts>

---

As a pleb I can think 1 of 2 things..

Something is wrong/off with some of my packages / versions / env / OS / setup OR, All the verifiers are lying and bitcoin's been compromised. Help me narrow down the possibilities.

I would like to go through this exercise properly and reproduce the same binaries.. but unsure of the best way forward.. what are the next steps to verifying the build for v0.20.1? I can't seem to reproduce the tarball advertised on bitcoincore . org and bitcoin . org. I can however make a reproducible build which makes me question a lot of things.

Would love if someone could go through their process from scratch and see if they can produce the same tarballs seen in this repo by these people. <https://github.com/bitcoin-core/gitian.sigs/blob/master/0.20.1-linux> /\*/ `bitcoin-core-linux-0.20-build.assert`'s.
And share their complete process with me. After reproducing myself I would be happy to make a current walkthrough on the method and update the Docs, which (links below) are dated and broken for nearly all methods.

NOTE: If this is indeed the defacto way to currently do reproducible builds.. it would be nice to have documentation that is precise and current. I understand Ginx is something being worked on.. but would be nice to have a current doc in the interim for people who wants to verify builds. I would also be very satisfied if someone pointed something I was missing.

**reference links:**  
<https://gitian.org/>  
<https://github.com/bitcoin-core/docs/blob/master/gitian-building.md>  
<https://github.com/bitcoin-core/docs/tree/master/gitian-building>  
<https://github.com/devrandom/gitian-builder/blob/master/README.md>  
<https://github.com/bitcoin/bitcoin/blob/master/contrib/gitian-build.py>

**reference github issue**  
<https://github.com/devrandom/gitian-builder/issues/235>

**Tags:** bitcoin-core, bitcoind, compiling, gitian, guix

> **Comments:**
> **username_1**: FYI, reported the problem at https://github.com/bitcoin/bitcoin/issues/20389  

---

## 2 answers

### Answer by username_3 on 2020-11-12. Score: 5 | Accepted answer

I've done a rebuild of 0.20.1 and I get the same results that you do. This would indicate that a build dependency has updated to produce slightly different results than the version that was in use at the time of the release. The build dependency versions are pinned unlike the actual software dependencies. IIRC this is common for gitian builds and attempting to rebuild even older releases will result in similar mismatches.

There is ongoing work to move to a different reproducible builds system called guix. This build system will pin the exact dependency versions, along with the option to build the entire toolchain from scratch. This should enable fully reproducible builds that can be replicated at any point in time.

---

Here is a more in depth explanation of this particular mismatch, copied from my response [in this GitHub issue](https://github.com/bitcoin/bitcoin/issues/20389#issuecomment-727527843).

If you look at the gitian build results, you will see a file named `bitcoin-0.20.1-x86_64-linux-gnu-debug.tar.gz`. If you untar this file, there will be `*.dbg` files, e.g. `bitcoind.dbg` and `bitcoin-qt.dbg`. These `*.dbg` files contain the debug data for their respective binaries, i.e. `bitcoind.dbg` contains debug data for `bitcoind`.

To ensure that you use the correct `dbg` file with the correct binary, gcc embeds a checksum of the `dbg` file within the binary itself. This means that `bitcoind` contains a checksum of `bitcoind.dbg`. This is to prevent attempting to debug `bitcoind` with another `dbg` file. For example, if you attempted to tell gdb that the `bitcoin-qt.dbg` file contained the debugging symbols for `bitcoind`, it would detect it does not and not attempt to load debugging symbols from `bitcoin-qt.dbg`.

The debugging symbols are initially compiled into the `bitcoind` binary, but later during the gitian build these are removed and placed into the `bitcoind.dbg` file. However because they are compiled into `bitcoind` initially, the debugging symbols have an effect on the build id that gcc embeds into the binary. The build ID is a hash of the compiled binary, including the debugging data.

So the end result is that the published binary contains two commitments to the debug symbols, but does not actually contain them itself.

You can read more about these separate debug files here: <https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html>

The crux of this issue is that the debug symbols from recent gitian builds differ from the debug symbols that were generated for the original release. This means that both the build ID and the debug symbol checksum that we find inside of `bitcoind` are different. This then results in the `bitcoind` hashes being different (as well as all of the other binaries). And of course that causes the tarfile hashes to be different.

---

Here is the diff of the binaries that diffoscope generates:

```
--- bitcoind
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│  
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│  
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│  
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ @@ -1,5 +1,5 @@
│  
│  Hex dump of section '.gnu_debuglink':
│    0x00000000 62697463 6f696e64 2e646267 00000000 bitcoind.dbg....
│ -  0x00000010 b25ceebb                            .\..
│ +  0x00000010 114d519d                            .MQ.

```

As you can see, there are only two differences here, one in the build ID, and one in the `.gnu_debuglink` section. From the documentation I linked earlier, we can see that this `.gnu_debuglink` section has the first line is the debug filename followed by enough 0 bytes to pad to a 4 byte boundary. The second line is the 4 byte CRC checksum. And it is this CRC checksum that differs.

So why do the debug symbols differ here? Again, diffoscope can help us a bit.

```
--- bitcoind.dbg
+++ /mnt/archive/bitcoin/bitcoin-binaries/0.20.1/bitcoin-0.20.1/bin/bitcoind.dbg
├── readelf --wide --notes {}
│ @@ -1,15 +1,15 @@
│  
│  Displaying notes found in: .note.ABI-tag
│    Owner                Data size     Description
│    GNU                  0x00000010    NT_GNU_ABI_TAG (ABI version tag)        OS: Linux, ABI: 3.2.0
│  
│  Displaying notes found in: .note.gnu.build-id
│    Owner                Data size     Description
│ -  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 6b464617f7f91fd270ac86f43ef4a58eeeedff19
│ +  GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)     Build ID: 3a439a31a5157ff7052ed310050df5643a02ea3f
│  
│  Displaying notes found in: .note.stapsdt
│    Owner                Data size     Description
│    stapsdt              0x00000036    NT_STAPSDT (SystemTap probe descriptors)        Provider: libstdcxx
│      Name: throw
│      Location: 0x00000000006e550d, Base: 0x000000000086d140, Semaphore: 0x0000000000000000
│      Arguments: 8@%rdi 8@%rsi
├── readelf --wide --debug-dump=info {}
│┄ error from `readelf --wide --debug-dump=info {}`:
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│┄ readelf: Error: /build/binutils/src/binutils-gdb/binutils/dwarf.c:1989: read LEB value is too large to store in destination variable
│ @@ -85052,36 +85052,36 @@
│      <29607>   DW_AT_decl_line   : 124
│      <29608>   DW_AT_decl_column : 16
│      <29609>   DW_AT_type        : <0x28761>
│      <2960d>   DW_AT_data_member_location: 12
│   <2><2960e>: Abbrev Number: 30 (DW_TAG_member)
│      <2960f>   DW_AT_name        : (indirect string, offset: 0x13aaf): __kind
│      <29613>   DW_AT_decl_file   : 108
│ -    <29614>   DW_AT_decl_line   : 148
│ +    <29614>   DW_AT_decl_line   : 128
│      <29615>   DW_AT_decl_column : 7
│      <29616>   DW_AT_type        : <0x287a6>
│      <2961a>   DW_AT_data_member_location: 16
│   <2><2961b>: Abbrev Number: 30 (DW_TAG_member)
│      <2961c>   DW_AT_name        : (indirect string, offset: 0x7a535): __spins
│      <29620>   DW_AT_decl_file   : 108
│ -    <29621>   DW_AT_decl_line   : 154
│ +    <29621>   DW_AT_decl_line   : 134
│      <29622>   DW_AT_decl_column : 3
│      <29623>   DW_AT_type        : <0x2879a>
│      <29627>   DW_AT_data_member_location: 20
...

```

There's a lot more output that I haven't included because it's pretty much all of the same.

Now this isn't terribly helpful, but we can see that for a bunch of functions, the line number for that function differs by 20 lines.

To get some more information, I used dwarfdump. This is what it says for the new build for the two functions I show in diffoscope (the functions are \_\_kind and \_\_spins).

```
0x0002960e:     DW_TAG_member
                  DW_AT_name    ("__kind")
                  DW_AT_decl_file       ("/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h")
                  DW_AT_decl_line       (148)
                  DW_AT_decl_column     (0x07)
                  DW_AT_type    (0x000287a6 "int")
                  DW_AT_data_member_location    (0x10)

0x0002961b:     DW_TAG_member
                  DW_AT_name    ("__spins")
                  DW_AT_decl_file       ("/usr/include/x86_64-linux-gnu/bits/thread-shared-types.h")
                  DW_AT_decl_line       (154)
                  DW_AT_decl_column     (0x03)
                  DW_AT_type    (0x0002879a "short int")
                  DW_AT_data_member_location    (0x14)

```

As you can see by the given file name, these functions come from libraries installed to the system. These appear to be headers for gcc's implementation of the c++ stdlib.

---

So what's happened is that libstdc++ has updated in Ubuntu. Whatever updates hapened have moved some code in some header files that Bitcoin Core includes in its use of the c++ stdlib. In turn, compiling with those updated headers results in different debug symbols because function declarations have moved in those header files. This then results in gcc computing a different build ID and a different CRC checksum for the debug symbols. This lastly results in the final binaries being slightly different, which causes the hashes to mismatch.

> **Comments:**
> **username_4**: I'm curious as to the choice of guix over Nix - do you have a link to any discussion on irc, github, etc. on why guix was chosen?  
> **username_3**: @username_4 From https://github.com/bitcoin/bitcoin/pull/15277: "Guix is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on bootstrappability and reproducibility which are attractive for security-sensitive projects like bitcoin." I don't think there has been much discussion about guix vs nix, just that guix is what someone has put the effort into doing, so it's what we get.  
> **username_0**: Thanks for the detailed info. If it's interesting I have cmp output here comparing both bitcoind files.  https://github.com/devrandom/gitian-builder/issues/235 Luke-jr let me know the files it looked like were in question were: .note.gnu.build-id .gnu_debuglink  @AndrewChow you think this is a design flaw or working as intended?  
> **username_3**: It is a design flaw of the gitian system, but the gitian system is also working as intended.  

---

### Answer by username_2 on 2020-11-12. Score: 1

As long as the underlying OS image is susceptible to changes, you will not get a reproducible build system that guarantees reproducibility over a long period of time. If the OS can update compiler toolchains, libc versions and whatever tools are used during a gitian build, changes in the future to the output are expected. Since this is the case since gitian is ubuntu based, and patches are still shipped to older ubuntu versions, this is to be expected after some time.


---

