Q: Strategy around saving application usage data for analysis and statistics We're in the process of figuring out a strategy for saving application usage data and later on compile reporting based on this data. 
A brief overview of the system: A C# web application with two type of users: Readers and Publishers. Publishers uploads documents in different categories and the Readers can open/download these documents. They can also search for Documents based on category and subscribe to documents in different categories.
The goal of the current project is start to collect data about the Readers usage of the system, what they search for, what they read/download and provide summery data for Publishers. 
We want to save data about each search performed by a Readers, to answer questions like "How many Readers filtered a search with category X or Y"? 
We also want to be able to report back aggregated data to Publishers and tell them "Your document appeared in X number of searches during the last month", "X people opened document Z the last month".
I'm trying to figure out how to approach this problem, here are some ideas


*

*We could save each search and it's filters as a entry in the database.

*When rendering a search result we could save a entry in the database to know that document X was included in a search result.

*We could also save entries in the same way when a Reader views/downloads a document.


If we would implement the strategy above that would mean that we'd have to write a lot. For a search result page I'm thinking something like 21 new entries - one entry for the details about the search-query and 20 entries for each "hit" for a document).
To avoid writing this to the db we could use some kind of message bus or in memory que to store the data out of the web request (to avoid slowing down the search experience).
Based on the above I have some questions:


*

*I'm I thinking in a good direction? Missing something? Or how could I approach this? Any recommended reading? Patterns or practises?

*If the above is a decent approach, would storing it in ie SQL Server would be appropriate? I'm a little concerned with this as I don't want the database to be busy writing these stats entries all the time. Is there another data store that would be a better fit?


A: Well, this sounds like a classic case for RabbitMq.Net. RabbitMq is ideal for processing a very very large number of events (to be processed in queues), when you don't really want to save all those lines to the db. All you want after all is to save some kind of cube of data (some kind of OLAP cube). That's a lot less data than the original lines. To give a simple example - Suppose you have 50,000 searches each day, and each of these produces 20 hits on average, you are looking at 1 million lines per day. But you only have 1000 publisher, each with say 50 publications. That's just 50,000 lines. And not for a single day. For all days. The only thing that will grow will be the amount of total hits (the updates on those lines). Here's a terrific guide to RabbitMq.Net - both hands on and theoretical
By far the best course on RabbitMq I've ever seen - and it's free
