<p>My solution was based on the one of @rombob.
I used S3 and created a daily S3 inventory file of the source bucket (which is replicated). Then I created a Athena DB and table using Glue (in CloudFormation). 
Then I implemented an S3 event which triggers a lambda. Every time a new manifest.checksum file was uploaded (= new inventory is finished). 
The lambda executes a query on the Athena table and checks if there are objects which <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/crr-status.html" rel="nofollow noreferrer">replication status</a> is FAILED. If there are then the lambda publishes to a SNS topic to send an email. </p>

CC BY-SA 4.0
2019-07-15T18:50:38.430Z
2019-07-15T18:50:38.430Z
lvthillo
<p>Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects.</p>

<p>Prerequisites</p>

<hr>

<ol>
<li><a href="https://docs.aws.amazon.com/AmazonS3/latest/user-guide/configure-inventory.html" rel="nofollow noreferrer">Enable inventory reports</a> on <code>src</code> (Source) and <code>dst</code>
(Destination) buckets</li>
<li><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/storage-inventory.html#storage-inventory-athena-query" rel="nofollow noreferrer">Create Athena tables</a> (<code>src_inventory</code>, <code>dst_inventory</code>) based on inventory reports for each bucket</li>
</ol>

<p>Monitor</p>

<hr>

<p>Run Athena query after both inventory reports generated e.g. on daily basis, can be executed from <a href="https://docs.aws.amazon.com/athena/latest/APIReference/API_StartQueryExecution.html" rel="nofollow noreferrer">Lambda</a>, <a href="https://github.com/anselmo/step_function_activity_patterns/blob/master/athena_query_pull_mode/fns/athena.js" rel="nofollow noreferrer">Step Function</a>, build server or any type of cron job.</p>

<p>Example of Athena query which should show number of missing objects on destination bucket at the time of generation of inventory reports:</p>

<pre><code>FROM src_inventory src 
LEFT JOIN dst_inventory dst 
ON src.key = dst.key 
WHERE dst.key is NULL
AND src.dt = '2018-11-11-08-00';
</code></pre>

<p>The result can be reported as CloudWatch metric and monitored with CloudWatch alarm. </p>

CC BY-SA 4.0
2018-11-14T20:10:08.953Z
2019-05-31T14:57:46.150Z
2019-05-31T14:57:46.150Z
rombob
<p>At this point using S3 inventory to monitor replication is the best solution for large buckets, and <a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/crr-status.html" rel="nofollow noreferrer">the solution suggested by AWS</a> themselves. Since AWS don't report on the status directly, any other solution is likely to be both expensive and time consuming when that many objects are involved. It seems like you're on the right (and only?) track.</p>

<p>An obvious idea would be to talk to AWS Support about it, ask them to add such a feature. But that's obvious.</p>

CC BY-SA 4.0
2019-02-11T17:55:51.913Z
2019-02-11T17:55:51.913Z
bgdnlp
<p>Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region.</p>

<p>The problem is that solution does not provide visibility on state for replication process, for example at the moment there's no way to easily monitor missing objects on destination or any possible permission issues that can interfere with the process and can result with replication not doing it's job, as well missing feedback for replication lag.</p>

<p>We found several tools similar to CRR Monitor <a href="https://aws.amazon.com/answers/infrastructure-management/crr-monitor/" rel="nofollow noreferrer">https://aws.amazon.com/answers/infrastructure-management/crr-monitor/</a>
but we didn't have good experience with it as we manage tens of millions of objects which can cost several thousand of dollars per month for CloudWatch events, as well as additional management for services required by solution (DinamoDB, Kinesis etc.) another layer of "monitoring the monitor" required.</p>

<p>For now we are using daily comparison of S3 objects inventory between source and destination using AWS Athena query, which also feels like a hack and it's not optimal for problem detection as inventory generated with daily frequency.</p>

<p>I hope to see some feedback, suggestions, ideas or experience for alternatives. Ideal solution would be managed or "serverless" solution which will improve visibility on the process and will provide fast detection of errors.</p>

<amazon-web-services><monitoring><amazon-s3>
How to monitor AWS S3 cross-region replication process?
rombob
# How to monitor AWS S3 cross-region replication process?

(Asked by: username_0 on 2018-10-04)

Amazon S3 has a cross-region replication which will handle copy of new/updated objects to additional region.

The problem is that solution does not provide visibility on state for replication process, for example at the moment there's no way to easily monitor missing objects on destination or any possible permission issues that can interfere with the process and can result with replication not doing it's job, as well missing feedback for replication lag.

We found several tools similar to CRR Monitor <https://aws.amazon.com/answers/infrastructure-management/crr-monitor/>
but we didn't have good experience with it as we manage tens of millions of objects which can cost several thousand of dollars per month for CloudWatch events, as well as additional management for services required by solution (DinamoDB, Kinesis etc.) another layer of "monitoring the monitor" required.

For now we are using daily comparison of S3 objects inventory between source and destination using AWS Athena query, which also feels like a hack and it's not optimal for problem detection as inventory generated with daily frequency.

I hope to see some feedback, suggestions, ideas or experience for alternatives. Ideal solution would be managed or "serverless" solution which will improve visibility on the process and will provide fast detection of errors.

**Tags:** amazon-web-services, monitoring, amazon-s3


---

## 3 answers

### Answer by username_1 on 2019-07-15. Score: 1 | Accepted answer

My solution was based on the one of @username_0.
I used S3 and created a daily S3 inventory file of the source bucket (which is replicated). Then I created a Athena DB and table using Glue (in CloudFormation).
Then I implemented an S3 event which triggers a lambda. Every time a new manifest.checksum file was uploaded (= new inventory is finished).
The lambda executes a query on the Athena table and checks if there are objects which [replication status](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr-status.html) is FAILED. If there are then the lambda publishes to a SNS topic to send an email.


---

### Answer by username_0 on 2018-11-14. Score: 2

Still no answers, and still no CRR monitoring in place on AWS, here's minimalistic solution, any feedback and suggestions are welcome. This monitor will only show which S3 objects are missing from destination, it will not provide visibility on versions of replicated objects.
Prerequisites


Enable inventory reports on src (Source) and dst
(Destination) buckets
Create Athena tables (src_inventory, dst_inventory) based on inventory reports for each bucket

Monitor

Run Athena query after both inventory reports generated e.g. on daily basis, can be executed from Lambda, Step Function, build server or any type of cron job.
Example of Athena query which should show number of missing objects on destination bucket at the time of generation of inventory reports:
FROM src_inventory src 
LEFT JOIN dst_inventory dst 
ON src.key = dst.key 
WHERE dst.key is NULL
AND src.dt = '2018-11-11-08-00';

The result can be reported as CloudWatch metric and monitored with CloudWatch alarm.


---

### Answer by username_2 on 2019-02-11. Score: 1

At this point using S3 inventory to monitor replication is the best solution for large buckets, and [the solution suggested by AWS](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr-status.html) themselves. Since AWS don't report on the status directly, any other solution is likely to be both expensive and time consuming when that many objects are involved. It seems like you're on the right (and only?) track.

An obvious idea would be to talk to AWS Support about it, ask them to add such a feature. But that's obvious.


---

