<p>I'm not sure a large code paste is the best way to get your example across (I'm unfamiliar with the library/API you're using for example). Assuming it's correct to summarize that you are looping, repeatedly doing <a href="https://ripple.com/wiki/JSON_Messages#account_tx" rel="nofollow">account_tx</a> RPC calls with at most a ten second delay; then that's not efficient even for a small number of accounts.</p>

<p>For a small-ish number of accounts, I'd recommend doing something like your loop <em>only</em> on restart to catchup from a last known ledger you've processed. (For example, after restarting your monitoring after an outage or disconnect). Once caught up it's better to use the <a href="https://ripple.com/wiki/JSON_Messages#subscribe" rel="nofollow">subscribe</a> RPC with the set of Ripple accounts you're interested in. The server will send you a notification anytime any transaction touches any of the accounts you list. This avoids polling and is far more efficient, especially during periods where the accounts are idle. (I'm only familar with using subscribe over a websocket, and it works very well for this even over days; I don't know what library support for this is like).</p>

<p>Once you go beyond a certain threshold of accounts you are trying to watch, especially if they are very active accounts, it may be easier to use the subscribe RPC to subscribe to all transactions and filter those for just the ones effecting the accounts you're interested in. Also, at that point it may become a good idea to do such queries against your own locally running rippled rather than rely on a public Ripple server.</p>

<p>The point at which it's best to switch from subscribing to a list of accounts versus subscribing to all transactions is unclear to me. For starters, there may be a limit to the number of accounts you can specify in a subscribe call; if so perhaps modify a locally running rippled to remove any such limit and have it do the filtering may be appropriate.</p>

CC BY-SA 3.0
2014-04-07T07:12:55.470Z
Thanks for replying. The RippleNetwork thing is just a thin wrapper around rippled websocket API. I aware the subscribe method, but how do you recover to the point where the monitor daemon fails? Repeated transactions are acceptable, while missing several is not.
Proton
CC BY-SA 3.0
2014-04-07T07:18:11.970Z
Subscribing all transactions and do filtering is also acceptable. But there is no "ledger_index_min" like account_tx provided, only live transaction streams.
Proton
CC BY-SA 3.0
2014-04-07T14:06:21.233Z
@Proton "but how do you recover to the point where the monitor daemon fails". I covered that in the second paragraph. That's where it would make sense to do a loop like the one you gave. You'd keep track of the highest ledger number you've synced with (either via the ledger field of each tx or if you happen to also want/need the ledger close stream you could use that). On restart you'd do the account_tx from that ledger forward, and once caught up with the current ledger you'd switch over to the subscribe stream for efficiency and lower latency rather than needing to poll for changes.
dchapes
CC BY-SA 3.0
2014-04-07T14:07:18.687Z
At least that's how I'd approach it, there are probably other ways.
dchapes
CC BY-SA 3.0
2014-04-06T13:37:08.983Z
2014-04-06T13:37:08.983Z
dchapes


<pre><code>rpc = RippleNetwork(app.config['RIPPLE_SERVER'])
poll = RippleNetwork(app.config['RIPPLE_SERVER'])
last_ledger = Variables.get('ripple_last_ledger')
poll.subscribe(streams=['ledger'])

while True:
    trans = []
    try:
        rst = rpc.account_tx(account=withdraw_wallet, ledger_index_min=last_ledger, limit=1000)
        trans.extend(rst['transactions'])
        while rst.get('marker') is not None:
            rst = rpc.account_tx(account=withdraw_wallet, ledger_index_min=last_ledger, marker=rst['marker'], limit=1000)
            trans.extend(rst['transactions'])

    except RippleException as e:
        if e.resp['error'] == 'lgrIdxsInvalid':
            log.debug('ledger_index_min too new, sleeping for 10s')
            time.sleep(10)
            continue

        raise

    for rec in trans:
        tx = rec['tx']
        log.debug('Got tx %s', tx['hash'])

        if tx['TransactionType'] != 'Payment':
            continue

        if tx['Destination'] != withdraw_wallet:
            continue

        log.info('Queue tx %s', tx['hash'])
        tube.put(str(tx['hash']))

    last_ledger = rst['ledger_index_max'] + 1
    Variables.set('ripple_last_ledger', last_ledger)
    db.session.commit()
    # Wait for next(or several) ledger close here.
</code></pre>

<p>Code above is my current implementation, but this method does not scale to thousands of accounts.</p>

<p>Is there any 'best practice'?</p>

CC BY-SA 3.0
2014-04-06T08:54:47.840Z
First, unless you're catching up on past transactions use subscribe instead of repeated account_tx calls. Second, for a large number of accounts it can be more efficient to just subscribe to the the full transaction stream and filter it down to the transactions you care about.
dchapes
CC BY-SA 3.0
2014-04-06T10:28:52.020Z
dchapes: why don't you add that as an answer?  I agree that if you're monitoring a significant number of accounts, it's best to get the full stream and filter.
Eric S
<ripple>
Ripple: How to monitor multiple accounts reliably and effectively?
Proton
# Ripple: How to monitor multiple accounts reliably and effectively?

(Asked by: username_0 on 2014-04-06)

```
rpc = RippleNetwork(app.config['RIPPLE_SERVER'])
poll = RippleNetwork(app.config['RIPPLE_SERVER'])
last_ledger = Variables.get('ripple_last_ledger')
poll.subscribe(streams=['ledger'])

while True:
    trans = []
    try:
        rst = rpc.account_tx(account=withdraw_wallet, ledger_index_min=last_ledger, limit=1000)
        trans.extend(rst['transactions'])
        while rst.get('marker') is not None:
            rst = rpc.account_tx(account=withdraw_wallet, ledger_index_min=last_ledger, marker=rst['marker'], limit=1000)
            trans.extend(rst['transactions'])

    except RippleException as e:
        if e.resp['error'] == 'lgrIdxsInvalid':
            log.debug('ledger_index_min too new, sleeping for 10s')
            time.sleep(10)
            continue

        raise

    for rec in trans:
        tx = rec['tx']
        log.debug('Got tx %s', tx['hash'])

        if tx['TransactionType'] != 'Payment':
            continue

        if tx['Destination'] != withdraw_wallet:
            continue

        log.info('Queue tx %s', tx['hash'])
        tube.put(str(tx['hash']))

    last_ledger = rst['ledger_index_max'] + 1
    Variables.set('ripple_last_ledger', last_ledger)
    db.session.commit()
    # Wait for next(or several) ledger close here.

```

Code above is my current implementation, but this method does not scale to thousands of accounts.

Is there any 'best practice'?

**Tags:** ripple

> **Comments:**
> **username_1**: First, unless you're catching up on past transactions use subscribe instead of repeated account_tx calls. Second, for a large number of accounts it can be more efficient to just subscribe to the the full transaction stream and filter it down to the transactions you care about.  
> **username_2**: username_1: why don't you add that as an answer?  I agree that if you're monitoring a significant number of accounts, it's best to get the full stream and filter.  

---

## 1 answers

### Answer by username_1 on 2014-04-06. Score: 0 | Accepted answer

I'm not sure a large code paste is the best way to get your example across (I'm unfamiliar with the library/API you're using for example). Assuming it's correct to summarize that you are looping, repeatedly doing [account\_tx](https://ripple.com/wiki/JSON_Messages#account_tx) RPC calls with at most a ten second delay; then that's not efficient even for a small number of accounts.

For a small-ish number of accounts, I'd recommend doing something like your loop *only* on restart to catchup from a last known ledger you've processed. (For example, after restarting your monitoring after an outage or disconnect). Once caught up it's better to use the [subscribe](https://ripple.com/wiki/JSON_Messages#subscribe) RPC with the set of Ripple accounts you're interested in. The server will send you a notification anytime any transaction touches any of the accounts you list. This avoids polling and is far more efficient, especially during periods where the accounts are idle. (I'm only familar with using subscribe over a websocket, and it works very well for this even over days; I don't know what library support for this is like).

Once you go beyond a certain threshold of accounts you are trying to watch, especially if they are very active accounts, it may be easier to use the subscribe RPC to subscribe to all transactions and filter those for just the ones effecting the accounts you're interested in. Also, at that point it may become a good idea to do such queries against your own locally running rippled rather than rely on a public Ripple server.

The point at which it's best to switch from subscribing to a list of accounts versus subscribing to all transactions is unclear to me. For starters, there may be a limit to the number of accounts you can specify in a subscribe call; if so perhaps modify a locally running rippled to remove any such limit and have it do the filtering may be appropriate.

> **Comments:**
> **username_0**: Thanks for replying. The RippleNetwork thing is just a thin wrapper around rippled websocket API. I aware the subscribe method, but how do you recover to the point where the monitor daemon fails? Repeated transactions are acceptable, while missing several is not.  
> **username_0**: Subscribing all transactions and do filtering is also acceptable. But there is no "ledger_index_min" like account_tx provided, only live transaction streams.  
> **username_1**: @username_0 "but how do you recover to the point where the monitor daemon fails". I covered that in the second paragraph. That's where it would make sense to do a loop like the one you gave. You'd keep track of the highest ledger number you've synced with (either via the ledger field of each tx or if you happen to also want/need the ledger close stream you could use that). On restart you'd do the account_tx from that ledger forward, and once caught up with the current ledger you'd switch over to the subscribe stream for efficiency and lower latency rather than needing to poll for changes.  
> **username_1**: At least that's how I'd approach it, there are probably other ways.  

---

