<p>ok figured it out! after about a week :-D.  I want to post this because I spent the last week learning the bitcoin core to output the intermittent values during the sighash to compare. Anyone else needing to really understand how the core works this will hopefully be really helpful. So first my specific problem: my output script was just completely incorrect, I think I got the locking and the unlocking script mixed up, which still shouldn't invalidate the signature, but then also you notice in the example the output is like:</p>
<pre><code>202cb20600000000 1976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac
</code></pre>
<p>well the first 0x19 is not part of the script public key (the way decode raw transaction displays it). So in the description where it says</p>
<blockquote>
<p>If the sighash type is neither SINGLE nor NONE, hashOutputs is the double SHA256 of the serialization of all output amount (8-byte little endian) with scriptPubKey (serialized as scripts inside CTxOuts);</p>
</blockquote>
<p>you hash with the length prefix 0x19. I was including the length in the overall transaction so it still decoded fine, but not in the way I hashed. Simple misunderstanding.</p>
<p><strong>Below is how I built and modified the core (0.21.1) to output the values to figure this out</strong></p>
<ol>
<li>I mostly followed this example <a href="https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests" rel="nofollow noreferrer">https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests</a>, basically installed most the dependencies from that list</li>
<li>export BDB_PREFIX='/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/db4'</li>
<li>./configure BDB_LIBS=&quot;-L${BDB_PREFIX}/lib -ldb_cxx-4.8&quot; BDB_CFLAGS=&quot;-I${BDB_PREFIX}/include&quot; --enable-debug</li>
<li>make</li>
<li>Notice the use of --enable-debug. I think this was needed to let you use gdb? I did occasionally get thread lock errors in the log file which would cause it to crash in debug mode when using the wallet api.</li>
<li>I created a separate bin folder with some sym links to the 4 main binary executables. then edited /etc/environment to append the path to that folder. then ran &quot;source /etc/environment&quot;. This lets you run the binaries from anywhere.</li>
<li>now you can run bitcoind -regtest -daemon. From here you can go into the source  code where the binaries were built and run gdb, and attach to the process.</li>
<li>Modify &lt;bitcoin_dir&gt;/src/script.interpreter.cpp At the top I wrote some basic function to take a few different variable types and print to hex. Apologies for the bad code, I am C person by trade, not C++. I created a separate logging file so i didn't mess with bitcoind's main log</li>
</ol>
<pre>
std::string hexhalftostr(unsigned char val)
{
    if (val &lt;= 9)
        return std::to_string(val);
    else if(val ==10)
        return "A";
    else if(val ==11)
        return "B";
    else if(val ==12)
        return "C";
    else if(val ==13)
        return "D";
    else if(val ==14)
        return "E";
    else if(val ==15)
        return "F";
    else return "";
}
void logbytes(unsigned char *data, int len)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    std::string ss = "";
    for (int i = 0;i &lt; len;i++)
    {
        char lsb, msb;
        lsb = 0x0f & data[i];
        msb = 0x0f & (data[i] &lt;&lt; 4);
        ss += hexhalftostr(msb) + hexhalftostr(lsb);
    }
    myfile &lt;&lt; ss;
    myfile.close();
}
void logbigint(uint256 data)
{
    logbytes(data.data(), 32);
}
void logint(unsigned int data)
{
    unsigned char arrayOfByte[4];
    for (int i = 0; i &lt; 4; i++)
        arrayOfByte[i] = (data &lt;&lt; (i * 8));
    logbytes(arrayOfByte, 4);
}
void loglong(unsigned long data)
{
    unsigned char arrayOfByte[8];
    for (int i = 0; i &lt; 8; i++)
        arrayOfByte[i] = (data &lt;&lt; (i * 8));
    logbytes(arrayOfByte, 8);
}
void logtext(std::string ss)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    myfile &lt;&lt; ss;
    myfile.close();
}

</pre>
<ol start="9">
<li>SignatureHash is the function where we can actually output them as they are being computed</li>
</ol>
<pre>
 logtext("\nnVersion:     ");
        logint(txTo.nVersion);
        logtext("\nhashPrevouts: ");
        logbigint(hashPrevouts);
        logtext("\nhashSquence:  ");
        logbigint(hashSequence);
        logtext("\noutpoint:  ");
        logbigint(txTo.vin[nIn].prevout.hash);
        logint(txTo.vin[nIn].prevout.n);
        //can't figure out how to grab the bytes
        ////logtext("\nscriptCode:  ");
        ////logbigint(scriptCode);
        logtext("\namount:  ");
        loglong(amount);
        logtext("\nnSequence:  ");
        logint(txTo.vin[nIn].nSequence);
        logtext("\nhashOutputs:  ");
        logbigint(hashOutputs);
        logtext("\nnLockTime:  ");
        logint(txTo.nLockTime);
        logtext("\nhashType:  ");
        logint(nHashType);
</pre>
<p>And the return of the function had to be replaced because the sha256 hash consumes the data so</p>
<pre><code> return ss.GetHash();
</code></pre>
<p>Becomes</p>
<pre><code> uint256 sighash = ss.GetHash();
 logtext(&quot;\nhashed image: &quot;);
 logbigint(sighash);
 logtext(&quot;\n&quot;);
 //return ss.GetHash();
 return sighash;
</code></pre>
<ol start="10">
<li>I would recomend setting a break point and stepping through &quot;EvalScript&quot; function. This is where all the magic happens, and is super good for educating yourself on how the actual bitcoin script runs, because each error code encompasses virtually unlimited mistakes, I am not sure there is any other way to debug your code without doing this.</li>
</ol>

CC BY-SA 4.0
2021-06-28T01:09:24.593Z
2021-06-28T01:09:24.593Z
noone392
<p>So I am trying to do my own witness signatures as described in bip143. I have code that correctly verifies the example signature, so I thought I had a hold of the basic signing and verification. I have code that can verify the signatures in bip143 correctly, and using the same code I verify this output correctly, so I am not sure how to determine what is going wrong. The error that I get is</p>
<pre><code>error code: -26
error message:
non-mandatory-script-verify-flag (Non-canonical DER signature)
</code></pre>
<p>I suspect that I am not reversing the byte order in something, but I am really not sure how to further diagnose this.</p>
<pre>
----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
<strong>dhashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 2cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d549e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3
der sig: 304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3 </strong>
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a8580247304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a301210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------
</pre>
<p>This is running on my regtest. I can provide any additional info that people might think would be helpful. I have spent many hours comparing every bit possible to all the examples before posting here. I know it says non connical der sig, but my researching, other people say that error happens when many different things go wrong. Also when I compare the signature to the core signed ones they are an exact match in format with the &quot;30440220&quot; then 32 bytes then 0220 then the other 32 bytes. The only thing that is different is the actual sig data. Plus I get the same error when using a native connical DER output.  Any help is greatly appreciated as always
<br>
-----UPDATE-----<br>
so the problem with the above script is that mainly it is not DER encoded correctly, meaning that if s or r most MSB &gt; 0x80 it treats it as an negative because it is a signed integer. In this example the MSB of S is 0xE8  which is greater then 0x80. bitcoin has a policy that it actually must be less than the ecdsa curve value &quot;n&quot;. First below is the code that corrects the problem, and I can verify the signature (but the error still exist in bitcoin core). Code to invert S in python:</p>
<pre><code>n=115792089237316195423570985008687907852837564279074904382605163141518161494337
s = bytearray(s_bytes)
sint = 0
cnt = 31
for bb in s:
    sint += bb &lt;&lt; cnt*8
    cnt -= 1
if sint &gt;= n/2:
    print(&quot;Negated&quot;)
    sint = n-sint
    s = bytearray(sint.to_bytes(32, byteorder='big'))
</code></pre>
<p>not to bad. And as for r the bitcoin core just re-signs until it is lower than 0x80. it is legal for it to be higher if the der encoder correctly pads the MSB). So did it the same way. below is a new transaction that has the problem corrected, though I still get the same error :-/.</p>
<pre>
----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
hashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 38d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a686be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
<strong>s before: be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
Negated
s after: 4187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
der sig: 3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521</strong>
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85802473044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d72679929201210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------</pre>
<p>Again, help is appreciated</p>

CC BY-SA 4.0
2021-06-22T01:08:57.577Z
That is not valid DER encoding. When the top (first) byte is >= 0x80, the represented number is interpreted as negative. When the value encoded is larger than 2^255, that means an additional 0 byte in front must be added. The Bitcoin network actually has an additional constraint, which requires the S value in the signature to be below n/2. This is very easy: when s >= n/2, negate it (replace with n-s).
Pieter Wuille
CC BY-SA 4.0
2021-06-22T01:30:13.573Z
Sorry I had to try it out. ok so when I was using the native canonical DER formatting it did sometimes have the leading 0 and was one longer. so to make sure I understand, basically when it is bigger than 0x80 instead of just adding in the 00 we instead negate the entire S ? so if s[0] > 0x80,  s = 0xFFFFFFFF...-s?
noone392
CC BY-SA 4.0
2021-06-22T02:01:08.287Z
@PieterWuille I tried what I thought was correct. no success yet. negating S when MSB is 1 and not adding the leading zero it won't validate in general now. How does bitcoin core know if you have inverted it or not?
noone392
CC BY-SA 4.0
2021-06-22T03:11:20.750Z
The reason for doing this is malleability: we don't want 3rd parties relaying a transaction to change it such that it txid changes but without invalidating it. It turns out that for ECDSA, if (r,s) is a valid signature, then so is (r,n-s). To avoid that malleation, Bitcoin Core has a policy rule that it only relays transactions with signatures whose s value is < n/2. It does not need to know whether it was negated or not, as both forms are equally valid ECDSA.
Pieter Wuille
CC BY-SA 4.0
2021-06-22T03:12:14.990Z
Note that n is a rather nontrivial number, it's around 2^256 - 1.5*2^128 (search for secp256k1 curve order), and R and S and encoded in big endian notation.
Pieter Wuille
CC BY-SA 4.0
2021-06-22T04:42:37.050Z
ok I got s-n and my test validation program can validate the signature still. For bitcoin core, what about r? if its to big for the max value, are we ok, just adding the leading 0 ? and following a "default" der scheme?
noone392
CC BY-SA 4.0
2021-06-22T04:46:09.920Z
Yes, R just has to be in range [0,n). Since there is a 50% chance that the value is below 2^255 however, you can save a byte often by just trying to sign again. That's what Bitcoin Core does, but just for financial reasons (fee scales with signature size); it's perfectly legal to have 33-byte R values.
Pieter Wuille
CC BY-SA 4.0
2021-06-22T17:19:39.250Z
@PieterWuille ok So I have updated the example with the corrections. it still decodes fine, and I can still validate the signature myself. sendrawtransaction still has the same error :-/
noone392
CC BY-SA 4.0
2021-06-22T17:45:20.210Z
What is your DER encoded signature now?
Pieter Wuille
CC BY-SA 4.0
2021-06-22T17:47:19.830Z
3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
I mean its a new signature because of the randomness. I think everything else should be the same. In the update I posted the original and the negated S just for completeness
noone392
CC BY-SA 4.0
2021-06-22T17:48:56.880Z
That looks correctly DER encoded. If you still get the same problem, I have no idea.
Pieter Wuille
CC BY-SA 4.0
2021-06-22T18:12:20.467Z
@PieterWuille alright well I appreciate the help/check. thank you
noone392
<segregated-witness><signature><raw-transaction><bip143>
Creating Witness Signature: non-mandatory-script-verify-flag (Signature must be zero for failed CHECK(MULTI)SIG operation)
noone392
# Creating Witness Signature: non-mandatory-script-verify-flag (Signature must be zero for failed CHECK(MULTI)SIG operation)

(Asked by: username_0 on 2021-06-22)

So I am trying to do my own witness signatures as described in bip143. I have code that correctly verifies the example signature, so I thought I had a hold of the basic signing and verification. I have code that can verify the signatures in bip143 correctly, and using the same code I verify this output correctly, so I am not sure how to determine what is going wrong. The error that I get is

```
error code: -26
error message:
non-mandatory-script-verify-flag (Non-canonical DER signature)

```

I suspect that I am not reversing the byte order in something, but I am really not sure how to further diagnose this.

```

----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
dhashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 2cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d549e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3
der sig: 304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a3 
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a8580247304402202cbb748d06a1b87c933a3118d8c2a5a0ab02111bae777a9e52ba8571eee2d5490220e80a6e4cef24a172e64d11aa7b83c8005fe6fc078307b9e663ef9eb14fce05a301210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------

```

This is running on my regtest. I can provide any additional info that people might think would be helpful. I have spent many hours comparing every bit possible to all the examples before posting here. I know it says non connical der sig, but my researching, other people say that error happens when many different things go wrong. Also when I compare the signature to the core signed ones they are an exact match in format with the "30440220" then 32 bytes then 0220 then the other 32 bytes. The only thing that is different is the actual sig data. Plus I get the same error when using a native connical DER output. Any help is greatly appreciated as always
  
-----UPDATE-----  
so the problem with the above script is that mainly it is not DER encoded correctly, meaning that if s or r most MSB > 0x80 it treats it as an negative because it is a signed integer. In this example the MSB of S is 0xE8 which is greater then 0x80. bitcoin has a policy that it actually must be less than the ecdsa curve value "n". First below is the code that corrects the problem, and I can verify the signature (but the error still exist in bitcoin core). Code to invert S in python:

```
n=115792089237316195423570985008687907852837564279074904382605163141518161494337
s = bytearray(s_bytes)
sint = 0
cnt = 31
for bb in s:
    sint += bb << cnt*8
    cnt -= 1
if sint >= n/2:
    print("Negated")
    sint = n-sint
    s = bytearray(sint.to_bytes(32, byteorder='big'))

```

not to bad. And as for r the bitcoin core just re-signs until it is lower than 0x80. it is legal for it to be higher if the der encoder correctly pads the MSB). So did it the same way. below is a new transaction that has the problem corrected, though I still get the same error :-/.

```

----unsigned transaction-------
020000000102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85800000000
-------end unsigned transaction------
dhash(prevouts)=02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
dhash(sequences)=ffffffff
dhash(outputs)=400d030000000000001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a0100000000140f54e16bd59da94840741b69ab9b9db99467a858
nversion:      02000000
hashPrevouts: 9ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd
hashSequence: 3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e70665044
outpoint:     02303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda4000000000
scriptCode:   1976a914992fda25fdf8447092aa223a34bedd3572173d8688ac
amount:       00f2052a01000000
nSequence:    ffffffff
hashOutputs   a54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c1
nLockTime     00000000
nHashType     01000000
pre hash image: 020000009ebc8589966e2dd13cd64af1835262c2d8e931b4388a8ebe76620814b4f407bd3bb13029ce7b1f559ef5e747fcac439f1455a2ec7c5f09b72290795e7066504402303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda40000000001976a914992fda25fdf8447092aa223a34bedd3572173d8688ac00f2052a01000000ffffffffa54e6a8744fa773a5f1c64d7f60d53efd69f8d76e85e7817970a87990874e4c10000000001000000
hashed image: 4f8fce636bb4a44aa6a241e9597bbe54fce3bfe3426a49a94c5b006d42eac007
rawsig: 38d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a686be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
s before: be784798307a123bdfefce7463b6e7203b69d18950ea95ae684dddb5a9bcaeaf
Negated
s after: 4187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
der sig: 3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292
script pub key: 992fda25fdf8447092aa223a34bedd3572173d86
pub key: 0392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b36521
----signed transaction-------
0200000000010102303e285bb228344346acb5292397a761103efcd0182c4cf95b4d01b2bdda400000000000ffffffff02400d03000000000016001453f0b90d7be69d4f5f1356908a9e12fef93561df74e0022a010000001600140f54e16bd59da94840741b69ab9b9db99467a85802473044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d72679929201210392ff36d0ae9f3a74c0483fd309ff9144972b1dce6d6dfe4d9de474c721b3652100000000
-------end signed transaction------
```

Again, help is appreciated

**Tags:** segregated-witness, signature, raw-transaction, bip143

> **Comments:**
> **username_1**: That is not valid DER encoding. When the top (first) byte is >= 0x80, the represented number is interpreted as negative. When the value encoded is larger than 2^255, that means an additional 0 byte in front must be added. The Bitcoin network actually has an additional constraint, which requires the S value in the signature to be below n/2. This is very easy: when s >= n/2, negate it (replace with n-s).  
> **username_0**: Sorry I had to try it out. ok so when I was using the native canonical DER formatting it did sometimes have the leading 0 and was one longer. so to make sure I understand, basically when it is bigger than 0x80 instead of just adding in the 00 we instead negate the entire S ? so if s[0] > 0x80,  s = 0xFFFFFFFF...-s?  
> **username_0**: @username_1 I tried what I thought was correct. no success yet. negating S when MSB is 1 and not adding the leading zero it won't validate in general now. How does bitcoin core know if you have inverted it or not?  
> **username_1**: The reason for doing this is malleability: we don't want 3rd parties relaying a transaction to change it such that it txid changes but without invalidating it. It turns out that for ECDSA, if (r,s) is a valid signature, then so is (r,n-s). To avoid that malleation, Bitcoin Core has a policy rule that it only relays transactions with signatures whose s value is < n/2. It does not need to know whether it was negated or not, as both forms are equally valid ECDSA.  
> **username_1**: Note that n is a rather nontrivial number, it's around 2^256 - 1.5*2^128 (search for secp256k1 curve order), and R and S and encoded in big endian notation.  
> **username_0**: ok I got s-n and my test validation program can validate the signature still. For bitcoin core, what about r? if its to big for the max value, are we ok, just adding the leading 0 ? and following a "default" der scheme?  
> **username_1**: Yes, R just has to be in range [0,n). Since there is a 50% chance that the value is below 2^255 however, you can save a byte often by just trying to sign again. That's what Bitcoin Core does, but just for financial reasons (fee scales with signature size); it's perfectly legal to have 33-byte R values.  
> **username_0**: @username_1 ok So I have updated the example with the corrections. it still decodes fine, and I can still validate the signature myself. sendrawtransaction still has the same error :-/  
> **username_1**: What is your DER encoded signature now?  
> **username_0**: 3044022038d610fa975440d09b501864185e656546015ee98dd8b5b58eb41be547b6a68602204187b867cf85edc42010318b9c4918de7f450b5d5e5e0a8d578480d726799292 I mean its a new signature because of the randomness. I think everything else should be the same. In the update I posted the original and the negated S just for completeness  
> **username_1**: That looks correctly DER encoded. If you still get the same problem, I have no idea.  
> **username_0**: @username_1 alright well I appreciate the help/check. thank you  

---

## 1 answers

### Answer by username_0 on 2021-06-28. Score: 1 | Accepted answer

ok figured it out! after about a week :-D. I want to post this because I spent the last week learning the bitcoin core to output the intermittent values during the sighash to compare. Anyone else needing to really understand how the core works this will hopefully be really helpful. So first my specific problem: my output script was just completely incorrect, I think I got the locking and the unlocking script mixed up, which still shouldn't invalidate the signature, but then also you notice in the example the output is like:

```
202cb20600000000 1976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac

```

well the first 0x19 is not part of the script public key (the way decode raw transaction displays it). So in the description where it says

> If the sighash type is neither SINGLE nor NONE, hashOutputs is the double SHA256 of the serialization of all output amount (8-byte little endian) with scriptPubKey (serialized as scripts inside CTxOuts);

you hash with the length prefix 0x19. I was including the length in the overall transaction so it still decoded fine, but not in the way I hashed. Simple misunderstanding.

**Below is how I built and modified the core (0.21.1) to output the values to figure this out**

1. I mostly followed this example <https://jonatack.github.io/articles/how-to-compile-bitcoin-core-and-run-the-tests>, basically installed most the dependencies from that list
2. export BDB\_PREFIX='/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/db4'
3. ./configure BDB\_LIBS="-L${BDB\_PREFIX}/lib -ldb\_cxx-4.8" BDB\_CFLAGS="-I${BDB\_PREFIX}/include" --enable-debug
4. make
5. Notice the use of --enable-debug. I think this was needed to let you use gdb? I did occasionally get thread lock errors in the log file which would cause it to crash in debug mode when using the wallet api.
6. I created a separate bin folder with some sym links to the 4 main binary executables. then edited /etc/environment to append the path to that folder. then ran "source /etc/environment". This lets you run the binaries from anywhere.
7. now you can run bitcoind -regtest -daemon. From here you can go into the source code where the binaries were built and run gdb, and attach to the process.
8. Modify <bitcoin\_dir>/src/script.interpreter.cpp At the top I wrote some basic function to take a few different variable types and print to hex. Apologies for the bad code, I am C person by trade, not C++. I created a separate logging file so i didn't mess with bitcoind's main log

```

std::string hexhalftostr(unsigned char val)
{
    if (val <= 9)
        return std::to_string(val);
    else if(val ==10)
        return "A";
    else if(val ==11)
        return "B";
    else if(val ==12)
        return "C";
    else if(val ==13)
        return "D";
    else if(val ==14)
        return "E";
    else if(val ==15)
        return "F";
    else return "";
}
void logbytes(unsigned char *data, int len)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    std::string ss = "";
    for (int i = 0;i < len;i++)
    {
        char lsb, msb;
        lsb = 0x0f & data[i];
        msb = 0x0f & (data[i] << 4);
        ss += hexhalftostr(msb) + hexhalftostr(lsb);
    }
    myfile << ss;
    myfile.close();
}
void logbigint(uint256 data)
{
    logbytes(data.data(), 32);
}
void logint(unsigned int data)
{
    unsigned char arrayOfByte[4];
    for (int i = 0; i < 4; i++)
        arrayOfByte[i] = (data << (i * 8));
    logbytes(arrayOfByte, 4);
}
void loglong(unsigned long data)
{
    unsigned char arrayOfByte[8];
    for (int i = 0; i < 8; i++)
        arrayOfByte[i] = (data << (i * 8));
    logbytes(arrayOfByte, 8);
}
void logtext(std::string ss)
{
    std::ofstream myfile;
    myfile.open ("/home/ubuntu/btc/bitcoinsource/bitcoin-0.21.1/special.log", std::ios_base::app);
    myfile << ss;
    myfile.close();
}

```

9. SignatureHash is the function where we can actually output them as they are being computed

```

 logtext("\nnVersion:     ");
        logint(txTo.nVersion);
        logtext("\nhashPrevouts: ");
        logbigint(hashPrevouts);
        logtext("\nhashSquence:  ");
        logbigint(hashSequence);
        logtext("\noutpoint:  ");
        logbigint(txTo.vin[nIn].prevout.hash);
        logint(txTo.vin[nIn].prevout.n);
        //can't figure out how to grab the bytes
        ////logtext("\nscriptCode:  ");
        ////logbigint(scriptCode);
        logtext("\namount:  ");
        loglong(amount);
        logtext("\nnSequence:  ");
        logint(txTo.vin[nIn].nSequence);
        logtext("\nhashOutputs:  ");
        logbigint(hashOutputs);
        logtext("\nnLockTime:  ");
        logint(txTo.nLockTime);
        logtext("\nhashType:  ");
        logint(nHashType);

```

And the return of the function had to be replaced because the sha256 hash consumes the data so

```
 return ss.GetHash();

```

Becomes

```
 uint256 sighash = ss.GetHash();
 logtext("\nhashed image: ");
 logbigint(sighash);
 logtext("\n");
 //return ss.GetHash();
 return sighash;

```

10. I would recomend setting a break point and stepping through "EvalScript" function. This is where all the magic happens, and is super good for educating yourself on how the actual bitcoin script runs, because each error code encompasses virtually unlimited mistakes, I am not sure there is any other way to debug your code without doing this.


---

