$\begingroup$

I've been giving your query some thought. In my opinion, the biggest challenge you face is in appropriately accounting for a very messy, complex dataset. In other words, as I understand it you've got an underlying black box algorithm dynamically, sequentially and uncontrollably spewing out level changes and rewards to players who can drop in and out at will. Given this, what would be ideal for your understanding would be to set up A/B type experiments to test and tease out the drivers of participation. Regrettably, it doesn't sound like that would be possible. On the other hand, you do have 365 days of historic information.

Would it be possible to decompose or classify this evolving, 365 day background history into a comprehensive set of binary (y/n) background metrics? Since I know so little about the data, I'm struggling with the specifics of what these background metrics might be. These metrics could then form a set of rules against which participant responses can be evaluated. E.g., for some unit of time -- a minute, an hour, a day, a week, whatever -- across and against this set of metrics, this is that period's profile. It sounds like lags in the metrics could also be useful. Next, as each metric is manipulated, e.g., switched on or off, evaluate participant response. For me, the biggest problem with this approach is that there are so many things going on at the same time, drilling down to very tiny, minute changes to gain insight could be challenging or even impossible.

Key target variables could be player participation (opting in or dropping out), duration (survival) of play, intensity of play, extent of rewards, and so on. One way to describe this approach is as a type of regression discontinuity design where everything is known and controlled up to the point at which a change is introduced. Needless to say, there would be a lot of discontinuities to evaluate. Alternatively, this approach could also amount to a kind of agent based model or historic simulation in the context of a massive contingency table matrix of data. Based on their responses to the changes, player profiles could be formed and their drivers (the metrics) evaluated.

It's always best to start with the simplest analyses, considering cross-sectional swaths of data and ignoring the sequential component, e.g., participation (opting in or dropping out) can be evaluated at this preliminary level of analysis. This should help clarify the information. Motifs or "states of play" capturing participant clusters or segments at an aggregate or global level can be developed. Then, additional levels of complication can be successively introduced as needed.

A few functional forms suggest themselves: logistic regression for the regression discontinuities, massive contingency tables, agent-based models, and so on. Once the sequential, time series component is reintroduced, there are lots of additional models that might work. Aggarwal and Reddy's book Data Clustering has an excellent overview of time series clustering methodologies. Given that "states of play" have been created, hidden markov models based on the transitions in those states might be helpful. Steve Scott's papers on his Google website (https://research.google.com/pubs/author57989.html) would be helpful here...or Oded Netzer's work on hierarchical, hidden markov models (here ... https://www0.gsb.columbia.edu/mygsb/faculty/research/pubfiles/2618/HMM%20of%20Customer%20Relationship%20Dynamics.pdf) Another approach that I particularly like that is not driven by the moments of any distribution is the information theoretic, complexity-based, permutation distribution method of Andreas Brandmaier (here ... https://www0.gsb.columbia.edu/mygsb/faculty/research/pubfiles/2618/HMM%20of%20Customer%20Relationship%20Dynamics.pdf) Finally, recurrent, deep learning neural networks could also be brought to bear as outlined in this excellent review by Lin and Tegmark, Critical Behavior from Deep Dynamics (here ... https://ai2-s2-pdfs.s3.amazonaws.com/5ba0/3a03d844f10d7b4861d3b116818afe2b75f2.pdf).

Anyway, those are a few thoughts. Hope they help.

* A few possible global metrics, later edit *

A few global metrics that could be used to create the motifs, segments or states of play could include: