<blockquote>
<p><strong>Block packets contain tx_in_sig_script, is this shared with tx data from tx packets?</strong></p>
</blockquote>
<p>All network packets follow the same serialisation for transactions. It is specified in <a href="https://en.bitcoin.it/wiki/Protocol_documentation#tx" rel="nofollow noreferrer">https://en.bitcoin.it/wiki/Protocol_documentation#tx</a>.</p>
<p>So far as I know, if you transmit a <code>getdata</code> request for transactions, you should receive exactly the same byte-stream for a transaction that you would find inside the byte stream for a retrieved block.</p>
<hr />
<blockquote>
<p><strong>I figured that I could parse the data in reverse after the lock time.</strong></p>
</blockquote>
<p>misery and madness lies way That<br />
regret and Failure<br />
recrimination-self and Disappointment<br />
disillusionment and Sadness<br />
frustration and futility into descent spiral steep A<br />
path this choose not Do<br />
sanity to back way the see to wits the retain still you while Retreat<br />
way the not is way That</p>
<hr />
<pre><code>blocks&gt; blockchain -file blk03448.dat
blk03448.dat     67 blk,  150687 tx,  130612 segwit,  383145 ins,  457851 outs, 1.1536802s

blocks&gt; blockchain -file blk03448.dat
blk03448.dat     67 blk,  150687 tx,  130612 segwit,  383145 ins,  457851 outs, 702.4941ms
</code></pre>
<p>So, disregarding disk IO, the most simple unoptimised code can parse 150687/0.702 = 214,654 transactions a second (4.6 µs/tx) on a fairly ordinary 5 year old PC <strong>when parsing stuff forwards</strong>.</p>

CC BY-SA 4.0
2023-03-04T14:31:28.990Z
2023-03-04T14:57:10.653Z
2023-03-04T14:57:10.653Z
RedGrittyBrick
<p>I'm trying to find the signature script that is in a tx message so I can better parse data. I have found that it is in there sometimes, but largely the tx packets don't contain anything in the block packet. How can this be?</p>

CC BY-SA 4.0
2023-03-04T00:01:40.380Z
No, transactions are self-contained. Can you give an example of a transaction hex that you fail to parse?
Pieter Wuille
CC BY-SA 4.0
2023-03-04T01:24:48.783Z
Is what you are seeing perhaps explained by native segwit inputs having a signature script of length 0? Your question sounds like an X-Y problem. Are you actually trying to solve a completely different problem, came up with your own approach to a solution and are now asking about that? What is the big-picture issue you are trying to solve?
Murch
CC BY-SA 4.0
2023-03-04T04:30:14.483Z
@Murch what I'm trying to do is narrow down the amount of TX data I'm parsing. I figured that I could parse the data in reverse after the lock time. Then save bytes until I find a bite that matches the length of the bites stored. Does this make sense to do?
bitcoinluvr6969
CC BY-SA 4.0
2023-03-04T04:42:30.590Z
Going reverse does not seem like a promising approach to me. Since you don’t know the length of the field until you’ve seen the length indicator, it’s not clear to me how you would know that you have found it. Even if you find a byte that matches the length of what you walked back so far, it could just be a random match inside of the field. I would start from the front.
Murch
CC BY-SA 4.0
2023-03-04T04:42:57.270Z
There is this document that I’ve been working on that aims to describe the fields of transactions. Perhaps it would be useful to you: https://github.com/Xekyo/bips/blob/d119d3bdee38784f2ca70301f58eaa4933b312d0/bip-tx-terminology.mediawiki#user-content-Anatomy_of_a_serialized_transaction
Murch
CC BY-SA 4.0
2023-03-04T07:36:10.133Z
@Murch how do I determine I'm at a sequence? You don't say much about it on github
bitcoinluvr6969
CC BY-SA 4.0
2023-03-04T11:12:39.657Z
The >>100000 tx/sec code I gave in answer to a previous question should work in all cases. Note Murch's comment about Segwit. Your question definitely lacks clarity - I suggest you edit it to include minimal runnable code, sample data, actual output and expected output. If your code is slow the usual answer is better algorithms or a better language (e.g. compiled rather than interpreted)
RedGrittyBrick
CC BY-SA 4.0
2023-03-04T12:46:19.613Z
The sequence can take any value. I don't see how it could be done going backwards. Going forwards, the length of the input script tells you. After reading the input script length, you know how many bytes to read for the input script. After that there are four bytes of sequence.
Murch
CC BY-SA 4.0
2023-03-04T12:56:28.300Z
It seems to me that most of the difficulty you're experiencing is due to your approach. The transaction data structure has a number of variable length fields or can have multiple elements of the same type. Where there is flexibility, a counter or length indicator tells you how many elements or bytes to expect. Parsing such a data structure from the back sounds like a terrible idea, if it's not completely infeasible. I don't understand why you're implementing this from scratch in the first place, but please do yourself the favor of going the direction where it tells you what to expect.
Murch
CC BY-SA 4.0
2023-03-04T13:01:01.247Z
If you want to look at a few more transactions nicely broken down, check out e.g. yogh.io.
Murch
CC BY-SA 4.0
2023-03-04T13:16:17.073Z
@bitcoinluvr6969 There is no way you can parse transaction data backwards, as every element is prefixed with its length, you need to start from the beginning. It would be helpful if you can edit your question with some actual transaction (or otherwise) data you're trying to parse, so we can clarify exactly what you're seeing, rather than needing to guess all your misconceptions.
Pieter Wuille
CC BY-SA 4.0
2023-03-04T16:31:18.497Z
@Murch and everyone. Thank you very much for the good information you all have been providing. I'm curious though, how does one distinguish the data murch made on github v. the data found in yogh.io? It is apparent that you cannot traverse the bytes the same way. Or am I wrong?
bitcoinluvr6969
CC BY-SA 4.0
2023-03-04T19:43:46.063Z
@bitcoinluvr6969 It's very unclear what you're talking about. There is just one transaction format, and all code and descriptions out there deal with it all the same (including murch's explanation, and yogh.io). If something is unclear **please post an actual example of something that is confusing to you**.
Pieter Wuille
CC BY-SA 4.0
2023-03-05T07:34:59.103Z
@PieterWuille The example murch linked me to that you and he created does not include the following at the beginning of the raw transaction: magic bytes, tx command, payload size, payload checksum, previous block, merkle root, timestamp, bits, nonce, coinbase transaction, among others. Or, am I (like usual) wrong? If they are in there, how are they derived?
bitcoinluvr6969
CC BY-SA 4.0
2023-03-05T13:30:45.483Z
It sounds like you're looking at (arbitrary?) messages in the P2P protocol, as they are sent on the wire (that's where magic, command, payload size, checksum come from). In "tx" messages specifically, the payload is a transaction. @Murch's link, and yogh.io, are just explaining the transaction structure itself, without its wrapping P2P message. And transactions occur in other places too, e
g. in a "block" message a block payload is sent, which consists of a block header followed by a tx count, and then a list of transactions, all following the same transaction structure.
Pieter Wuille
CC BY-SA 4.0
2023-03-05T18:04:24.453Z
So you mean that transactions are sent in multiple places? So, I'm sorry to beat this into the ground but I just want to be thorough, and thank you for taking the time to explain this stuff to me. So, a block contains tx data, but the tx command also contains tx data? Which one of them is sent by multiple peers? Is it the block message? I only ever see one tx message being sent out.
bitcoinluvr6969
CC BY-SA 4.0
2023-03-05T21:43:03.937Z
@bitcoinluvr6969 It sounds to me like you need to read up on far more details about how the Bitcoin protocol works before diving into how the actual serialization works. Going that deep is out of scope for the comment section here.
Pieter Wuille
<transactions><block>
Block packets contain tx_in_sig_script, is this shared with tx data from tx packets?
bitcoinluvr6969
# Block packets contain tx_in_sig_script, is this shared with tx data from tx packets?

(Asked by: username_0 on 2023-03-03)

I'm trying to find the signature script that is in a tx message so I can better parse data. I have found that it is in there sometimes, but largely the tx packets don't contain anything in the block packet. How can this be?

**Tags:** transactions, block

> **Comments:**
> **username_1**: No, transactions are self-contained. Can you give an example of a transaction hex that you fail to parse?  
> **username_2**: Is what you are seeing perhaps explained by native segwit inputs having a signature script of length 0? Your question sounds like an X-Y problem. Are you actually trying to solve a completely different problem, came up with your own approach to a solution and are now asking about that? What is the big-picture issue you are trying to solve?  
> **username_0**: @username_2 what I'm trying to do is narrow down the amount of TX data I'm parsing. I figured that I could parse the data in reverse after the lock time. Then save bytes until I find a bite that matches the length of the bites stored. Does this make sense to do?  
> **username_2**: Going reverse does not seem like a promising approach to me. Since you don’t know the length of the field until you’ve seen the length indicator, it’s not clear to me how you would know that you have found it. Even if you find a byte that matches the length of what you walked back so far, it could just be a random match inside of the field. I would start from the front.  
> **username_2**: There is this document that I’ve been working on that aims to describe the fields of transactions. Perhaps it would be useful to you: https://github.com/Xekyo/bips/blob/d119d3bdee38784f2ca70301f58eaa4933b312d0/bip-tx-terminology.mediawiki#user-content-Anatomy_of_a_serialized_transaction  
> **username_0**: @username_2 how do I determine I'm at a sequence? You don't say much about it on github  
> **username_3**: The >>100000 tx/sec code I gave in answer to a previous question should work in all cases. Note username_2's comment about Segwit. Your question definitely lacks clarity - I suggest you edit it to include minimal runnable code, sample data, actual output and expected output. If your code is slow the usual answer is better algorithms or a better language (e.g. compiled rather than interpreted)  
> **username_2**: The sequence can take any value. I don't see how it could be done going backwards. Going forwards, the length of the input script tells you. After reading the input script length, you know how many bytes to read for the input script. After that there are four bytes of sequence.  
> **username_2**: It seems to me that most of the difficulty you're experiencing is due to your approach. The transaction data structure has a number of variable length fields or can have multiple elements of the same type. Where there is flexibility, a counter or length indicator tells you how many elements or bytes to expect. Parsing such a data structure from the back sounds like a terrible idea, if it's not completely infeasible. I don't understand why you're implementing this from scratch in the first place, but please do yourself the favor of going the direction where it tells you what to expect.  
> **username_2**: If you want to look at a few more transactions nicely broken down, check out e.g. yogh.io.  
> **username_1**: @username_0 There is no way you can parse transaction data backwards, as every element is prefixed with its length, you need to start from the beginning. It would be helpful if you can edit your question with some actual transaction (or otherwise) data you're trying to parse, so we can clarify exactly what you're seeing, rather than needing to guess all your misconceptions.  
> **username_0**: @username_2 and everyone. Thank you very much for the good information you all have been providing. I'm curious though, how does one distinguish the data username_2 made on github v. the data found in yogh.io? It is apparent that you cannot traverse the bytes the same way. Or am I wrong?  
> **username_1**: @username_0 It's very unclear what you're talking about. There is just one transaction format, and all code and descriptions out there deal with it all the same (including username_2's explanation, and yogh.io). If something is unclear **please post an actual example of something that is confusing to you**.  
> **username_0**: @username_1 The example username_2 linked me to that you and he created does not include the following at the beginning of the raw transaction: magic bytes, tx command, payload size, payload checksum, previous block, merkle root, timestamp, bits, nonce, coinbase transaction, among others. Or, am I (like usual) wrong? If they are in there, how are they derived?  
> **username_1**: It sounds like you're looking at (arbitrary?) messages in the P2P protocol, as they are sent on the wire (that's where magic, command, payload size, checksum come from). In "tx" messages specifically, the payload is a transaction. @username_2's link, and yogh.io, are just explaining the transaction structure itself, without its wrapping P2P message. And transactions occur in other places too, e g. in a "block" message a block payload is sent, which consists of a block header followed by a tx count, and then a list of transactions, all following the same transaction structure.  
> **username_0**: So you mean that transactions are sent in multiple places? So, I'm sorry to beat this into the ground but I just want to be thorough, and thank you for taking the time to explain this stuff to me. So, a block contains tx data, but the tx command also contains tx data? Which one of them is sent by multiple peers? Is it the block message? I only ever see one tx message being sent out.  
> **username_1**: @username_0 It sounds to me like you need to read up on far more details about how the Bitcoin protocol works before diving into how the actual serialization works. Going that deep is out of scope for the comment section here.  

---

## 1 answers

### Answer by username_3 on 2023-03-04. Score: 3

> **Block packets contain tx\_in\_sig\_script, is this shared with tx data from tx packets?**

All network packets follow the same serialisation for transactions. It is specified in <https://en.bitcoin.it/wiki/Protocol_documentation#tx>.

So far as I know, if you transmit a `getdata` request for transactions, you should receive exactly the same byte-stream for a transaction that you would find inside the byte stream for a retrieved block.

---

> **I figured that I could parse the data in reverse after the lock time.**

misery and madness lies way That  
regret and Failure  
recrimination-self and Disappointment  
disillusionment and Sadness  
frustration and futility into descent spiral steep A  
path this choose not Do  
sanity to back way the see to wits the retain still you while Retreat  
way the not is way That

---

```
blocks> blockchain -file blk03448.dat
blk03448.dat     67 blk,  150687 tx,  130612 segwit,  383145 ins,  457851 outs, 1.1536802s

blocks> blockchain -file blk03448.dat
blk03448.dat     67 blk,  150687 tx,  130612 segwit,  383145 ins,  457851 outs, 702.4941ms

```

So, disregarding disk IO, the most simple unoptimised code can parse 150687/0.702 = 214,654 transactions a second (4.6 µs/tx) on a fairly ordinary 5 year old PC **when parsing stuff forwards**.


---

