StreamingBandit
Maurits Kaptein
Jules Kruijswijk
This paper introduces StreamingBandit, a (back-end) solution for developing adaptive and personalized persuasive systems. Creating successful persuasive applications requires a combination of design, social science, and technology. StreamingBandit contributes to the required technology by providing a platform that can be used to adapt persuasive technologies in real-time and at large scales. We first introduce the design philosophy of StreamingBandit using a running example and highlight how a large number of adaptive persuasive systems can be regarded as solutions to (contextual) multi-armed bandit problems: a type of problem that StreamingBandit was built to address. Subsequently, we detail several scenarios of the use of StreamingBandit to create adaptive persuasive systems and detail its future developments.
article.tex
\documentclass[11pt]{amsart}

\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{bbm}
\usepackage{natbib}
\usepackage{listings}
\usepackage{url}
\lstset{language=Python, numbers=left}
\makeatletter
\renewcommand\bibsection%
{
  \section*{\refname
    \@mkboth{\MakeUppercase{\refname}}{\MakeUppercase{\refname}}}
}
\makeatother

\usepackage{listings,textcomp}
\lstset{basicstyle=\footnotesize\ttfamily}
%\lstset{framextopmargin=50pt,frame=bottomline}

\usepackage{color}
\definecolor{grey}{gray}{0.9}

\title{\texttt{StreamingBandit}: Developing Adaptive Persuasive Systems.}
\author{Maurits Kaptein}
\address{Tilburg University, Tilburg, the Netherlands. Statistics and Research Methods.}
\email{m.c.kaptein@uvt.nl}
\author{Jules Kruijswijk}
\address{Radboud University, Nijmegen, the Netherlands. Artificial Intelligence.}
\date{}

\begin{document}


\begin{abstract}
This paper introduces \texttt{StreamingBandit}, a (back-end) solution for developing adaptive and personalized persuasive systems. Creating successful persuasive applications requires a combination of design, social science, and technology. \texttt{StreamingBandit} contributes to the required technology by providing a platform that can be used to adapt persuasive technologies in real-time and at large scales. We first introduce the design philosophy of \texttt{StreamingBandit} using a running example and highlight how a large number of adaptive persuasive systems can be regarded as solutions to (contextual) multi-armed bandit problems: a type of problem that \texttt{StreamingBandit} was built to address. Subsequently, we detail several scenarios of the use of \texttt{StreamingBandit} to create adaptive persuasive systems and detail its future developments.
\end{abstract}

\maketitle

%\section{Introduction}\label{sec:Introduction}

Researchers in the persuasive technology field have demonstrated the effectiveness and utility of persuasive applications in diverse domains such as health-care, energy reduction, and interactive marketing \citep[see, e.g.,][]{kukkonen2009, Kaptein2011, Fogg2002}. It is striking to see that applications and technologies that were researched by persuasive technology scholars half a decade ago are now commonplace: for example, devices that track physical activity of users and provide interactive feedback are commercially available on a large scale \citep[e.g.,][]{Consolvo2008, Kaptein2012f}, and well-researched feedback methods -- such as goal setting -- have found numerous practical applications \citep{Lacroix2008a, Lacroix2009, Oinas-Kukkonen2009}. As such, the field has made a noticeable mark on society. 

However, there is an aspect of persuasive technologies that has long been advocated by scholars, but has by-and-large not left the research realm; this is the basic notion that persuasive technologies should ``deliver the right message, at the right time, to the right user'' \citep[see, e.g.,][]{Hirsh2012a, Oinas-Kukkonen2009, Kaptein2011}. Although there is a common understanding that persuasive technologies \emph{should} be made both adaptive and personalized, creating such systems is challenging. The successful development of adaptive persuasive systems requires a combination of design, social science, and technology. In this paper we contribute to the latter: we introduce \texttt{StreamingBandit}, a platform that supports the ``logic and reasoning'' of adaptive or personalized persuasive systems. \texttt{StreamingBandit} is available open-source to those wishing to create adaptive persuasive systems. In this paper we detail the design rationale of the platform, document its core functionality, and present several use-cases.

Section \ref{sec:2} introduces a formalization of adaptive persuasive systems that is the starting point for \texttt{StreamingBandit} using a running example of an application to promote physical activity which we coin \emph{RunSmart}. Section \ref{sec:introstrm} introduces the platform and details how RunSmart can be implemented. In Section \ref{sec:additional} we discuss a number of additional features of the platform, and in Section \ref{sec:scenarios} we present additional use cases. Finally, we discuss the limitations of the current project, and encourage readers to contribute to the further development of the platform.

\section{Formalizing adaptive persuasive systems}
\label{sec:2}

To detail the design choices made in \texttt{StreamingBandit} we first introduce the formalism that we have adopted to create the platform. We will use a running example of an adaptive application designed to promote physical activity coined RunSmart:

\medskip
\emph{RunSmart is an application that encourages users to work out more. The system is composed of a smart-phone that can provide user feedback, and a (water resistant) accelerometer which allows the application to measure user activity. Every day the application sets a goal for the user. This goal is either a number of kilometers to run that day, or a number of kilometers to swim. Subsequently, the application measures the performed activities. RunSmart is adaptive based on the weather conditions: RunSmart will propose indoor swimming on rainy days, and outdoor running on sunny days. RunSmart is also personalized: the recommended goal is personal for each user. Since goals that are set too low are likely to lead to low activity levels, and goals that are set too high are likely demotivating and thus will also result in low activity levels, RunSmart tries to set a personal, challenging, goal.}
\medskip

RunSmart can be formalized such that the technical challenges of adapting the activity to the weather conditions and personalizing the goal can be solved quite easily using \texttt{StreamingBandit}. We introduce the following notation:
\begin{itemize}
\item An index of the interactions $t=1, \dots, t=T$, where in our running example $t=1$ is the time the first goal is set by RunSmart and $t=T$ the last ($T$ is likely undefined at design time).
\item The \emph{context} $x_t \in \mathcal{X}_t$ where $\mathcal{X}$ is a set of variables describing the current state of the application. For RunSmart the relevant context vector $x$ consists of only two variables ($x^{(1)}$ and $x^{(2)}$), of which one describes the weather (``rainy'' or ``sunny''), and one identifies the current user. Note that we use \emph{super}scripts to identify elements of the context vector, and \emph{sub}scripts to identify time points.
\item The \emph{action} $a_t \in \mathcal{A}_t$ where $\mathcal{A}$ is a set of possible actions the application can take. In RunSmart the action vector consists of two variables, ($a^{(1)}$ and $a^{(2)}$); one denoting the activity (``running'' or ``swimming''), and one denoting the goal in kilometers.
\item The \emph{reward} $r_t$ is a (function of the) measured response at that point in time. In our running example this is the activity of the user as measured using the accelerometer. For simplicity we assume that RunSmart directly measures the number of kilometers that was swam or ran.
\item A \emph{policy} $\Pi \colon x_1,\dots,x_{t'},a_1,\dots,a_{t'-1},r_1, \dots, r_{t'-1}  \to a_{t'}$, which is a mapping from all possible interactions (their contexts, actions, and rewards) up to some point in time $t=t'$ to the next action $a_{t'}$. In our example this is the ``rule'' that is used to select the next activity and goal (at time $t=t'$) given all the previous observations.%\footnote{We quote the word ``rule'' since this is often interpreted as a deterministic mapping. In \texttt{StreamingBandit} this is not necessarily the case.}
\end{itemize}

The above provides a formalization of the sequential choices made by RunSmart in such a way that these can be used directly in \texttt{StreamingBandit}. In short, \texttt{StreamingBandit}, provides a platform that allows users to implement different policies: different ``rules'' to assign new actions based on the historical interactions and the current context. %Hence, \texttt{StreamingBandit} can be used to implement the ``rules'' that map the current context to suggested actions.

Note that the above formalization of the adaptive persuasive system in terms of
context, actions, and reward, is known as a contextual multi-armed bandit
problem (cMAB, or MAB for the simpler version without a context)
\citep{berry1985bandit, Whittle1980, Agrawal2014}. The cMAB formalization is a
reinforcement learning problem that is studied in mathematics and computer
science as a model for decision making in uncertainty, where the values of the actions need to be learned sequentially \citep{sutton1998introduction}. In the MAB problem the experimenter needs to balance trying out actions to learn about their associated reward, and selecting those actions that she believes to lead to high rewards. In the cMAB problem the experimenter is, prior to making the decision, confronted with additional information, the context, which might influence the reward associated with certain actions but cannot be manipulated. The experimenter is looking for a policy $\Pi$ which maximizes the cumulative reward $R(t) = \sum_{t=1}^T r_t | x_t, a_t$. A preferred policy can be conceived as the decision ``rule'' that selects actions given a context which produce the highest cumulated reward. For RunSmart we are looking for a policy that suggests goals (the actions), which give rise to the most active lifestyle of the user (the reward), given the specific user and the weather. 

\texttt{StreamingBandit} formalizes the challenge of designing adaptive persuasive systems as a cMAB problem, and allows designers to implement a policy on a webserver that can be integrated in the persuasive application. To ensure the scalability of \texttt{StreamingBandit} we have made the design choice to split the implementation of a policy into two steps. To do so, we assume that at time $t=t'$, the (expected) effect of the choice $a_{t'}$ given context $x_{t'}$ on the reward $r_{t'}$ can be summarized into a model: $r = f( a, x ; \theta_{t'} )$. Given this assumption, we can identify the following two steps of a policy:
\begin{enumerate}
\item The \emph{summary} step: In each summary step $\theta_{t'-1}$ is updated by the new information $\{x_{t'},a_{t'},r_{t'}\}$. Thus, $\theta_{t'} = g(\theta_{t'-1}, x_{t'},a_{t'},r_{t'})$ where $g()$ is some update function. Effectively, all the prior data, $x_1,\dots,x_t'$, $a_1,\dots$, $a_{t'}$, $r_1, \dots, r_{t'}$ are thus \emph{summarized} into $\theta_{t'}$ and the model $f()$, which jointly is assumed to have a much lower dimension then the data itself. This choice for a summary at each time point makes that the computation time is bounded by the dimension of $\theta$.\footnote{Note that this forces one to implement an online policy \citep{Michalak2012}. While perhaps not intuitively clear, this forces users of \texttt{StreamingBandit} to write summary schemes that are computationally bounded and scalable.}
\item The \emph{decision} step: In the decision step, the model $r = f( a, x_{t'} ; \theta_{t'} )$ is evaluated for the current context and the possible actions. Then, the recommended action at time $t'$ is selected. Note that one could naively think that $a_{max} = arg max_a f( a, x_{t'} ; \theta_{t'} )$ would be the best action to choose. However, this ignores the uncertainty in both $f()$ and $\theta$.
\end{enumerate}

The above might sound a bit theoretical and detached from the design proces of adaptive persuasive systems. However, the summation step and decision step are easily related to our RunSmart example: in the \emph{summation} step, RunSmart learns which goal leads to the highest reward based on the historical data. For example, a very simple model for RunSmart could be that the highest rewards is achieved when a goal that is 10\% higher then the average exercise length of a user is suggested to that user. In this case the summary step merely updates the average exercise length of a user each time new data enters: for example, one specific user on average runs 5 km and swims 1 km. These \emph{summaries} are the elements of $\theta$ for this user and are updated when newly measured activity data is received. Second, when a \emph{decision} needs to be made, the weather is inspected (contained in $x_t^1$), the summary ($\theta_{t'}$) is retrieved, and a decision is made: If it is raining ($x_t^1 = "rain"$) the user is recommended to go for a 1.1 km long swim ($\bar{x}_{swim} *1.1$). Thus, $a^1 = "swim", a^2 = 1.1$. Note that both the model and the decision rule presented here are extremely simplistic; the model for the ``best'' goal should to be learned from the data. Also, the uncertainty in the estimate of the average sport lengths need to be incorporated, and integrated in the decision rule.

\section{Introducing \texttt{StreamingBandit} }
\label{sec:introstrm}

In this section we introduce \texttt{StreamingBandit} and demonstrate how it can be used to implement the adaptation and personalization of RunSmart. Figure \ref{fig:1} presents an overview of the platform and it's main functionality. \texttt{StreamingBandit} is a python 3 application that runs a Tornado webserver (see \path{http://www.tornadoweb.org/en/stable/}) and which discloses a REST API that facilitates the implementation of the \emph{summary} and \emph{decision} steps as described above. The two main REST calls are:
\begin{itemize}
\item The \emph{decision} call:
\begin{lstlisting}[frame=none, numbers=none, backgroundcolor=\color{grey}] 
.../HOST/EXPID/getaction.json?key=EXPKEY&context={}
\end{lstlisting}
where \texttt{.../HOST} is the address where the \texttt{StreamingBandit} server is running, EXPID and EXPKEY are the ID and key of the current application (see Section \ref{sec:step1}), and the variable context contains a JSON object encoding the context $x_t$ (e.g., \texttt{\{"weather":"rainy", "userid":12\}}). The call returns a JSON formatted object containing the selected action given the policy which depends on the current summarized data $\theta_t$.\footnote{JSON is selected as the format for passing data from and to the server because of its omnipresence in web based applications}
\item The \emph{summarize} call: 
\begin{lstlisting}[frame=none, numbers=none, backgroundcolor=\color{grey}] 
.../HOST/EXPID/setreward.json?key=EXPKEY&context={}
    &action={}&reward={}
\end{lstlisting} where the context $x_t$, action $a_t$ and reward $r_t$ at a point in time are used to update $\theta_t$.
\end{itemize}
\texttt{StreamingBandit} allows users to create a new ``experiment'' to enable
the above two calls, and allows users to write custom \texttt{python 3} scripts
that implement the \emph{summary} and \emph{decision} steps. 

\begin{figure}[h!]
  \centering
    \includegraphics[width=.8\textwidth]{Figure1}
    \label{fig:1}
      \caption{Schematic overview of the core functionality of \texttt{StreamingBandit}, including two example REST API calls for the RunSmart application.}
\end{figure}

\subsection{Configuring  \texttt{StreamingBandit}}

Configuring \texttt{StreamingBandit} consists of three steps; first, one creates a new experiment which initializes up the associated REST calls. Second, one implements the \emph{summary} step logic in a custom \texttt{python} script which can be done using the web-based front-end of \texttt{StreamingBandit}. Finally, one implements the \emph{decision} step. We explain each in detail below.

\subsubsection{Step 1: Setup an experiment}
\label{sec:step1}

Step one is setting up a new experiment. Figure \ref{fig:2} presents a number of screenshots of the \texttt{StreamingBandit} front-end that can be used to setup new experiments by navigating to
\begin{lstlisting}[frame=none, numbers=none, backgroundcolor=\color{grey}] 
.../HOST/management.html
\end{lstlisting} 
The user is asked to provide a name for the experiment, and to provide the code for the summary and decision steps (see explanation below). After creating the experiment, it receives an experiment ID and key that need to be used in the subsequent REST calls. After creating the adaptive goal setting experiment it received \texttt{id=1} and \texttt{key=36207d46df} (see lower right panel), now enabling calls to (e.g.,)
\begin{lstlisting}[frame=none, numbers=none, backgroundcolor=\color{grey}] 
.../HOST/1/getAction.json?key=36207d46df&context={}
\end{lstlisting}  

\begin{figure}[h!]
  \centering
    \includegraphics[width=.95\textwidth]{Figure2}
    \label{fig:2}
      \caption{Screen shots of the front-end of \texttt{StreamingBandit}. Upper left is the welcome page (\texttt{http://HOST}). After clicking "management" one can specify a name for the experiment (upper right), and specify the code for the summary and decision steps (lower left). After the experiment is created it receives an id and key (lower right).}
\end{figure}

\subsubsection{Step 2: Implement the summary step}

Next, the actual logic needs to be implemented. For illustration purposes we continue with the RunSmart application, and focus on a \emph{summary} step which updates the average distance traveled for each user independently for both swimming and running. We enable this functionality by writing a custom script which, given a context, an action, and a reward, computes the averages. Consider the call:
\begin{lstlisting}[frame=none, numbers=none, backgroundcolor=\color{grey}] 
.../HOST/1/setreward.json?key=36207d46df&
    context={"weather":"sunny", "userid":12}&
    action={"type":run, "distance":6}&reward={"km":8}
\end{lstlisting}
which denotes that user number 12 on a sunny day received the goal to run for 6km (the action $a_t$), and ended up running 8 kilometer (the reward, $r_t$). We now update the average run length of user 12 that is stored in $\theta$ to incorporate the new information. To do so, we add the following summary code:

\begin{lstlisting}[frame=none]
import libs.base as base
key = "weather-uid"
value = self.context["weather"] + str(self.context["userid"])
average = base.Mean(self.get_theta(key=key, value=value))
average.update(self.reward["km"])
self.set_theta(average, key=key, value=value)
\end{lstlisting}

Any standard \texttt{python3} code can be used to define the summary step. However in the above code snippet we make use of a number of the functionalities implemented in \texttt{StreamingBandit}; these features are documented in detail in the manual that is available at \path{http://mkaptein.github.io/streamingbandit/}.

The code above first imports a separate toolkit from \texttt{libs.base} that is shipped with \texttt{StreamingBandit}. Our use of toolkits enable third-party developers to add functionality to \texttt{StreamingBandit}. Next, the current average for the current user and the current weather (both encoded in the \texttt{context} object, see lines 2-3) are retrieved in line 4, and it is casted as a ``base.Mean'' object. This object is updated with the number of kilometers ran (the reward) in line 3, and finally the new average for this user and this weather is stored.

\subsubsection{Step 3: Implement the decision step}

For the \emph{decision} step the code that is used is again standard \texttt{python3} code, possibly augmented with \texttt{StreamingBandit} functionality. For our example application the REST call to retrieve a new goal is:
\begin{lstlisting}[frame=none, numbers=none, backgroundcolor=\color{grey}] 
.../HOST/1/getAction.json?key=36207d46df&
   context={"weather":"sunny", "userid":12}
\end{lstlisting}
and the following code implements the response logic:
\begin{lstlisting}[frame=none]
import libs.base as base
key = "weather-uid"
value = self.context["weather"] + self.context["userid"]
average = base.Mean(self.get_theta(key=key, value=value))
self.action['distance'] = average.get_value() * 1.1
if self.action['distance'] == 0:
    self.action['distance'] = 1
self.action['type'] = "run" if 
   self.context["weather"] == "sunny" else "swim"
\end{lstlisting}
%\todo{Edit text to reflect changes in get and set theta!}

Here, line 4 again retrieves the average for the current user and context (as in the previous step), and line 5 computes the suggestion by adding 10\% to this average. Line 6-7 set an initial goal of 1 kilometer if no data is present. Finally, Line 8-9 add, based on the context, the suggested activity type. The \texttt{self.action} object is automatically returned and hence the server response will be a JSON object containing the variables "distance" and "type". 

Step 1--3 implement the server-side of the development of adaptive persuasive systems. Once the experiment is setup the two REST API calls are available. The smart phone application can now be integrated. For many smart-phone and web applications implementing the two REST calls and handling the responses will be straightforward since HTTP REST and JSON are common standards. 

Note that the three step process presented here implements a very rudimentary version of RunSmart's logic: the weather is tied one-to-one to the activity, the data of each user is treated independently, the goal setting model (adding 10\%) is fixed, and no attention is given to the uncertainty in any of the obtained estimates. In real applications one would likely use a much more sophisticated (e.g., hierarchical) model to model the relationship between the actions, the context, and the rewards, and use some method of balancing exploration and exploitation \citep[e.g., Thompson sampling, ][]{Chapelle2011}. One of the advantages of \texttt{StreamingBandit} however is the fact that once setup, one can easily extend and improve the logic of the adaptation. 

\section{Additional features}
\label{sec:additional}

In this section we briefly describe a number of features of \texttt{Streaming\-Bandit} that might be of interest for designers of persuasive systems. First, \texttt{StreamingBandit} comes with a number of different libraries (such as \texttt{libs.base} mentioned above) to enable streaming processing of data. The \texttt{libs.base} toolkit allows researchers to easily update counts, means, proportions, variances, and co-variances in data streams. Similarly, the \texttt{libs.lm} library allows fitting of linear regression models in data streams. The \texttt{libs.thompson} library implements Thompson sampling \citep[e.g.,][]{Chapelle2011} for the Bernoulli bandit problem. Currently we are working on developing more toolkits, and invite others to contribute toolkits together with documented examples. Next to the toolkits, \texttt{StreamingBandit} comes with a number of default implementations for simple policies such as $\epsilon$-first (the AB test), Thompson sampling, and a number of others. These are available through the management console and can easily be expanded (see also Section \ref{sec:scenarios}). 

\texttt{StreamingBandit} also allows logging of all the calls that are made and all the data that is collected. Within the custom summary and decision steps explicit calls to \texttt{self.log\_data(\{\})} can be made to store JSON objects encoding (e.g.), the current calls, the time, etc. This allows developers of adaptive persuasive systems to run post-hoc analysis on the outcomes of an evaluation. The collected logs, as well as the current state of $\theta$ are available through separate REST calls and can thus also be integrated into applications. Similarly, the management front-end, as depicted in Figure \ref{fig:2}, is fully detached from the core and its functionality is enabled through a number of separate REST calls: hence developers can easily develop their own front-end to setup experiments.

We have tried to make \texttt{StreamingBandit} both scalable and secure: the management console can easily be protected using signed cookies, and the REST calls to the the core \emph{summary} and \emph{decision} steps are protected using the experiment id and key combination. Scalability is ensured by a) using state-of-the art technologies in the development of \texttt{StreamingBandit} such as Tornado (as a base for the webserver), and Redis (an extremely fast, in-memory data-base system), and b) by forcing, by design, the use of online (streaming) analysis.

Finally, \texttt{StreamingBandit} allows ``nested'' experiments: hence, the
summary or decision steps of an experiment can call, using
\texttt{self.run\_experiment(id)}, the summary or decision code of
\emph{another} experiment. This extremely powerful feature allows one to
implement complex logic comparing -- using e.g., an AB test (see Section \ref{AB:test} -- multiple methods of adaptation or personalization). This latter functionality for example would enable designers of the RunSmart app to develop multiple goal setting algorithms, and compare the effectiveness of each within a single experiment. The nesting of experiments makes \texttt{StreamingBandit} a flexible platform to test, evaluate, and iterate different adaptation and personalization attempts.

\section{Possible scenarios}
\label{sec:scenarios}

In this section we describe a number of possible uses of \texttt{StreamingBandit}. This section aims to highlight the versatility \texttt{StreamingBandit} has to offer for designers of adaptive persuasive systems; it does not aim to give a full overview of the implementation of these scenarios.

\subsection{AB testing}
\label{AB:test}

A simple application of \texttt{StreamingBandit} is AB testing: comparing two versions of an interface in a randomized controlled experiment. In its simplest case there is no context (thus, this is a MAB problem, not cMAB), there are two possible actions $a_t \in \{A,B\}$, and the reward is a click on a webpage ($r_t \in \{0,1\}$). The summarize step of such an AB test can be implemented as follows:
\begin{lstlisting}[frame=all]
import libs.base as base
prop = base.Proportion(self.get_theta(key="version", 
   value=self.action["version"]))
prop.update(self.reward["click"])
self.set_theta(prop, key="version", value=self.action["version"])
\end{lstlisting}
This code updates and stores the click proportion of the different versions (the actions) of the application. The decision step is given by:
\begin{lstlisting}[frame=all]
import libs.base as base
propl = base.List(self.get_theta(key="version"),
  base.Proportion, ["A", "B"])  
if propl.count() > 1000:
  self.action["version"] = propl.max()
else
  self.action["version"] = propl.random()
\end{lstlisting}
%McK still not sure whether this would work, since the set_theta still needs a name of the self.action....
Which implements that up to 1000 interactions ($t=1000$) randomly version A and B are suggested. After 1000 interactions, the version with the highest click through proportion is suggested. Note that not providing a value in the \texttt{self.get\_theta} call results in a list of the data for all possible values.
%Note the use of \texttt{all\_action=True} to retrieve a \emph{list} of proportions for all possible actions (versions of the interface) as opposed to a single proportion as in the summary step.

\subsection{Dynamic AB testing}

The AB test presented above, while commonplace, ignores the uncertainty in the estimated proportions. Instead of adopting a fixed time exploration period (the experiment, $t < 1000$), and then moving to an exploitation period (selecting the version that performed best during the trial), one can also balance exploration and exploitation in a more refined fashion.\footnote{Space limitations in this paper do not allow us to dig into the details of the exploration-exploitation trade-off, or the possible methods to address this issue. For more information we recommend \citep{Macready1998}.} This is also coined \emph{dynamic} AB testing, and one selects a version, "A" or "B", proportional to the belief that that version has the highest reward \citep[this is a policy known as Thompson sampling, see][]{THOMPSON1933}. To implement a dynamic AB test the summary step remains as above, and the decision step is changed to:
\begin{lstlisting}[frame=all]
import libs.thompson as thmp
propl = thmp.BBThompsonList(self.get_theta(key="version"), 
   Proportion, ["A", "B"])
self.action["version"] = propl.thompson()
\end{lstlisting}
This example highlights that the summary step and the decision step are loosely coupled, and that sometimes new policies can be created by combining existing summary and decision steps.

\subsection{Learning a linear model}

The RunSmart example application as discussed in Sections \ref{sec:2} \&
\ref{sec:introstrm} can easily be altered by changing the goal-setting method.
One way of doing so would be to remove the assumption that an increase of 10\%
over the current average distance is optimal and replace it by a goal that is
learned from the rewards. We could for example define $\delta$ as the
difference between the average distance $\bar{x}$ and the set goal $g$, and learn how $\delta$ relates to the actual reward using a simple linear model $r = \beta_0 + \delta \beta_1 + \delta^2 \beta_2 +\epsilon$ where $\epsilon \sim \mathcal{N}(0,\sigma^2)$. The value of the coefficients, $\beta_0,\dots,\beta_2$, of this simple model can be learned from the data. This can be done using the following code in the summary step:
\begin{lstlisting}[frame=all]
import libs.base as base
import libs.lm as lm
key = "weather-uid"
value = self.context["weather"] + str(self.context["userid"])
model = lm.Model(self.get_theta(key=key, value=value))
mean = base.Mean(self.get_theta(name="mean", key=key, value=value))
d = mean - self.action["km"]
model.update(self.reward["km"], [d, d^2])
mean.update(self.reward["km"])
self.set_theta(model, key=key, value=value)
self.set_theta(average, name="mean", key=key, value=value)			
\end{lstlisting}

The code above will fit the linear model in a data stream, and the coefficients
will be available in the decision step using \texttt{model.get\_coefs()} and can be used to determine the optimal goal.

It has to be noted that the examples provided above are relatively simple: they
use small or no context, and simple updating models. This somewhat obscures the
true potential of \texttt{StreamingBandit} to select actions in very complex
settings at very large scales. However, we hope that the above examples
introduced the main ideas behind \texttt{StreamingBandit} and encourages interested readers to develop their own policies.

\section{Limitations and Conclusions}
\label{sec:lim}

\texttt{StreamingBandit} is still work in progress: Although we have recently released the first stable version of the application on GitHub, we are still developing additional toolkits, examples, and documentation. Also, we are currently using \texttt{StreamingBandit} in the evaluation of adaptive persuasive technologies; we hope to be able to report on actual field trials powered by \texttt{StreamingBandit} in the near future. However, we think the current version is mature enough to share with the Persuasive Technology community, and to encourage others to use the application. We are actively seeking feedback to improve the application and maximize its use.

We are currently aware of a number of limitations: first, the application is currently run and tested only on a single core. While parallelization for large scale is easy at the level of experiments, this will be harder within experiments; this is an obvious next step in ensuring scalability. Also, the toolkits could be greatly extended and improved; for example, many methods to address exploration-exploitation trade-off in various settings have been developed in the machine learning and statistical literature and could be included.

In this paper we have briefly introduced \texttt{StreamingBandit} and presented relatively simple examples. However, we hope the current expose is sufficient to a) communicate the design philosophy behind \texttt{StreamingBandit}, and b) raise the interest of those developing adaptive or personalized persuasive systems wishing to easily implement their adaptation logic: this is exactly where \texttt{StreamingBandit} can be of help. The latest version of  \texttt{StreamingBandit} can be downloaded from \path{https://github.com/MKaptein/streamingbandit}.

\bibliographystyle{apa} 
\bibliography{library}

\end{document}



article.bbl
\begin{thebibliography}{}

\bibitem[\protect\astroncite{Agrawal}{2014}]{Agrawal2014}
Agrawal, S. (2014).
\newblock {Thompson Sampling for Contextual Bandits with Linear Payoffs}.

\bibitem[\protect\astroncite{Berry and Fristedt}{1985}]{berry1985bandit}
Berry, D.~A. and Fristedt, B. (1985).
\newblock {\em {Bandit Problems: Sequential Allocation of Experiments}}.
\newblock Springer.

\bibitem[\protect\astroncite{Chapelle and Li}{2011}]{Chapelle2011}
Chapelle, O. and Li, L. (2011).
\newblock {An Empirical Evaluation of Thompson Sampling}.
\newblock In Shawe-Taylor, J., Zemel, R.~S., Bartlett, P., Pereira, F. C.~N.,
  and Weinberger, K.~Q., editors, {\em Advances in Neural Information
  Processing Systems}, pages 2249----2257.

\bibitem[\protect\astroncite{Consolvo et~al.}{2008}]{Consolvo2008}
Consolvo, S., Libby, R., Smith, I., Landay, J.~A., McDonald, D.~W., Toscos, T.,
  Chen, M.~Y., Froehlich, J., Harrison, B., Klasnja, P., LaMarca, A., and
  LeGrand, L. (2008).
\newblock {Activity sensing in the wild}.
\newblock In {\em Proceeding of the twenty-sixth annual CHI conference on Human
  factors in computing systems - CHI '08}, page 1797, New York, New York, USA.
  ACM Press.

\bibitem[\protect\astroncite{Fogg}{2002}]{Fogg2002}
Fogg, B.~J. (2002).
\newblock {\em {Persuasive Technology: Using Computers to Change What We Think
  and Do}}.
\newblock Morgan Kaufmann.

\bibitem[\protect\astroncite{Hirsh et~al.}{2012}]{Hirsh2012a}
Hirsh, J.~B., Kang, S.~K., and Bodenhausen, G.~V. (2012).
\newblock {Personalized Persuasion: Tailoring Persuasive Appeals to Recipients'
  Personality Traits}.
\newblock {\em Psychological Science}, (April).

\bibitem[\protect\astroncite{Kaptein et~al.}{2012}]{Kaptein2011}
Kaptein, M.~C., de~Ruyter, B., Markopoulos, P., and Aarts, E. (2012).
\newblock {Tailored Persuasive Text Messages to Reduce Snacking.}
\newblock {\em Transactions on Interactive Intelligent Systems}, 2(2):10----35.

\bibitem[\protect\astroncite{Kaptein and van Halteren}{2012}]{Kaptein2012f}
Kaptein, M.~C. and van Halteren, A. (2012).
\newblock {Adaptive Persuasive Messaging to Increase Service Retention}.
\newblock {\em Journal of Personal and Ubiquitous Computing}, 17(6):1173--1185.

\bibitem[\protect\astroncite{Lacroix et~al.}{2009}]{Lacroix2009}
Lacroix, J., Saini, P., and Goris, A. (2009).
\newblock {Understanding user cognitions to guide the tailoring of persuasive
  technology-based physical activity interventions}.
\newblock In Chatterjee, S. and Dev, P., editors, {\em Persuasive Technology,
  Fourth International Conference, PERSUASIVE 2009}, volume 350 of {\em ACM
  International Conference Proceeding Series}, page~9, Claremont, California,
  USA. ACM.

\bibitem[\protect\astroncite{Lacroix et~al.}{2008}]{Lacroix2008a}
Lacroix, J., Saini, P., and Holmes, R. (2008).
\newblock {The relationship between goal difficulty and performance in the
  context of a physical activity intervention program}.
\newblock In {\em Proceedings of the 10th international conference on Human
  computer interaction with mobile devices and services - MobileHCI '08}, page
  415, New York, New York, USA. ACM Press.

\bibitem[\protect\astroncite{Macready and Wolpert}{1998}]{Macready1998}
Macready, W.~G. and Wolpert, D.~H. (1998).
\newblock {Bandit problems and the exploration/exploitation tradeoff}.
\newblock {\em IEEE Transactions on Evolutionary Computation}, 2(1):2--22.

\bibitem[\protect\astroncite{Michalak et~al.}{2012}]{Michalak2012}
Michalak, S., DuBois, A., DuBois, D., Wiel, S.~V., and Hogden, J. (2012).
\newblock {Developing Systems for Real-Time Streaming Analysis}.
\newblock {\em Journal of Computational and Graphical Statistics},
  21(3):561--580.

\bibitem[\protect\astroncite{Oinas-kukkonen and
  Harjumaa}{2009}]{Oinas-Kukkonen2009}
Oinas-kukkonen, H. and Harjumaa, M. (2009).
\newblock {Persuasive Systems Design : Key Issues , Process Model , and System
  Features}.
\newblock {\em Design}, 24.

\bibitem[\protect\astroncite{Sutton and Barto}{1998}]{sutton1998introduction}
Sutton, R.~S. and Barto, A.~G. (1998).
\newblock {\em {Introduction to reinforcement learning}}.
\newblock MIT Press.

\bibitem[\protect\astroncite{Thompson}{1933}]{THOMPSON1933}
Thompson, W.~R. (1933).
\newblock {On the Likelihood that one Unknown Probability Exceeds Another in
  view of the Evidence of two Samples}.
\newblock {\em Biometrika}, 25(3-4):285--294.

\bibitem[\protect\astroncite{Tuomas and Oinas-Kukkonen}{2009}]{kukkonen2009}
Tuomas, L. and Oinas-Kukkonen, H. (2009).
\newblock {The persuasiveness of Web-based alcohol interventions}.
\newblock In {\em 9th IFIP Conference on e-Business, e-Services, and
  e-Society}, pages 164--176.

\bibitem[\protect\astroncite{Whittle}{1980}]{Whittle1980}
Whittle, P. (1980).
\newblock {Multi-armed bandits and the Gittins index}.
\newblock {\em Journal of the Royal Statistical Society Series B
  Methodological}, 42(2):143--149.

\end{thebibliography}
