project: Moodle
summary: Project Inspire Phase II proposal
description: This Epic holds the proposed plan for Project Inspire Phase II, in which the community will be empowered to create, test, and share models, without requiring PHP coding or Moodle developer skills. The following issues are of the highest priority: All bugs identified in 3.4 and 3.5 testing cycles Metadata additions: Site , Activity and Course (necessary to improve model fits and generalizability) Allow targets to override access control to predictions (otherwise we cannot deliver insights to anyone except instructors and managers) Allow model configurations to be imported and exported (development already in progress) Implement UI for creating and editing learning analytics based on reports - this is the key new feature, which will make it possible for non-developers to construct models and share them, including model training data. This will lower the priority of many of the other feature requests in the backlog. Alternatively, is a proposal to implement a UI for creating and editing simple learning analytics models based only on events. For the use of analytics in large courses, especially MOOCs, the following is critical: Allow bulk actions to notifications from learning analytics Finally, Display accuracy of models more prominently and with more detail is an important feature to support researchers developing and testing models.

1.summary: Display accuracy of models more prominently and with more detail
description: Currently the accuracy of models is reported only in the Log action menu. Model accuracy is a key aspect of learning analytics, and needs to be displayed more prominently. Accuracy varies with time-splitting methods, but only one time-splitting method can be active for a model at a time, so the accuracy level for the currently used time-splitting method should be displayed on the list of models. More detail should be available about the accuracy of each model, as well. One simple way to display this information is by using a ROC curve plotting the true positive rate of the model predictions against the false positive rate: Display the following details for each model: Contingency distribution (graph) ROC curve (graph) Area under curve (number) Contingency table and diagnostics: (note that most of these are simple calculations of a small number of basic values) True positive n Power Recall/Sensitivity (TPR) False positive n Type I error Fall-out (FPR) False negative n Type II error Miss rate (FNR) True negative n Specificity (TNR) Predicted positive n Prevalence Precision (PPV) LR+ Predicted negative n False omission (FOR) LR- Condition positive n Condition negative n Total n Diagnostics: Accuracy False discovery rate Negative prediction value DIagnostic odds ratio F1 score Detail of indicators: Selected indicator ROC curves (graph) with control to select how many indicators to display Model effect size (standardized) Indicator graphs: Indicator significance (p-value) Standardized effect sizes with confidence intervals for all indicators _Alternative charts, such as the detection error tradeoff graph or zROC could be used as the header graph instead of a ROC. Usability studies should determine which kind of display would be easiest for most users to understand and use in decision-making._ _Not that the ROC curve is only applicable for binary predictions, but at present these are the only prediction models supported by Moodle learning analytics. This could be extended to REC or RROC curves once we extend support for other types of predictions._
2.summary: Enhance enrolment record to include "expected start" and "expected end" dates
description: In "rolling enrollment" courses, there may be no single course start date. Because of this, the due dates for activities could be different for each student. suggests that the due dates for activities be allowed to be set as deltas from the student enrollment date, but the date the student was enrolled in the course may not be their expected start date. It could be beneficial to modify the enrollment record so that it contains both an enrolled date and an "expected start" date. Activity deadlines would be set based on the "expected start" date, rather than the enrollment date. Similarly, the enrollment record could contain two ending dates, one switching the enrollee to read-only access, the other preventing any access to the course. Any "end" dates for enrollment could be calculated based on the "expected start" date. It is also possible that one might want to set due dates (and course enrollment expected start dates) relative to the enrollment end date. This is key to supporting learning analytics for rolling-enrollment courses.
3.summary: Improve Predictions report
description: The current report of insights generated by learning analytics is sparse (see attached image) and could be more immediately informative. Currently the report consists of sections per prediction, and each section contains two columns: Name (or other sample entity) Actions menu Please add: likelihood of prediction Selection checkbox Bulk action function per prediction type Under Actions, please add "Review messages" as a way to quickly jump to the message history with the user the prediction is about (if it is about a user).
4.summary: Improve prediction details report
description: The current prediction details report generated by learning analytics could be more informative. Currently the report consists of a header with the name of the prediction, the name of the user (or other sample entity) the prediction is about, a menu of actions, and the date of the prediction. The date range covered by the prediction is in a long text form which is difficult to read. (See attached screenshot.) Below the header, indicators and calculated values are displayed in two columns. The rows are color-coded by the calculated values of each indicator. The spacing of the rows is loose, so it is difficult to get an overview of the status of all indicators at once. Please add to header: likelihood of prediction Under Actions, please add "Review messages" as a way to quickly jump to the message history with the user the prediction is about (if it is about a user). For each indicator, please add: A link to help text for that indicator Weight value in the model for that indicator Number of times that indicator has occurred for this sample (if it is an event)
5.summary: Improve analytics models display and administration
description: This is a proposal to make the management and display of the list of analytics models easier to read and more informative. In this proposal, the current list of analytics models would be slightly simplified (removing the indicator display and truncating the width of the "insights" drop-down), and the ability to add, duplicate, edit, and delete models using one of several potential model editors (e.g. or ) would be implemented. It would be possible to export, share, and import models, perhaps as xml or JSON files, including training data. This display would also highlight model training status and accuracy and usage calculations, to aid in evaluation of models.
6.summary: Invalid site elements report needs summary
description: The "Invalid site elements" report can go to multiple pages, but does not give any hint as to how many pages there will be. There is also no way to quickly determine the reasons for the invalid elements. The following enhancements are recommended: As with all analytics pages, please ensure that breadcrumbs are displayed and work. Display pager that indicates how many pages in total are in the report Provide summary on first page listing: How many courses are invalid to train Proportional breakdown of reasons those courses are invalid How many courses are invalid to get predictions Proportional breakdown of reasons those courses are invalid Other enhancements that would be helpful include display of additional information, e.g. categories, start/end dates, visibility, filters to view only courses which satisfy criteria (e.g. reason for invalidity), dates, categories, etc., ability to adjust number of elements per page of report, ability to sort by different criteria, etc. For reasons which could be corrected, e.g. course does not have an end time, provide link to a bulk editing tool to adjust end times, etc. (see related issue ) Allow elements to be "acknowledged" so they can be filtered from the report, so only new invalid elements can be displayed.
7.summary: Provide ability to edit course metadata in bulk
description: Several metadata fields attached to courses affect learning analytics and other reports, and some of these have been recently introduced or may have been used inconsistently on a site in the past. This feature would allow the site administrator or manager to filter courses by criteria (e.g. category, substrings of fullname or shortname, metadata that was filled e.g. startdate) and select courses for bulk update of any universal course fields/settings, including: Start date End date Visibility Add/Remove competency Course format Gradebook visibility Completion tracking Group mode Tags All metadata referenced in (Add Course metadata to support learning analytics) Instructional mode (online vs. hybrid vs. F2F) online activity Absolute/relative dates Expected time to complete Course level Subject area
8.summary: Provide tools to conduct forensics on analytics when prediction fails
description: The learning analytics system makes predictions about educational outcomes such as student success or course effectiveness. Ideally, the analytics model will be highly accurate in its predictions. However, two kinds of errors should be reported and investigated: Type I or "False positive," when the model predicts that an outcome will occur but it fails to occur (e.g. a student is predicted to be at risk of dropping out, but the student continues in the course until completion) and Type II or "False negative," when the model fails to predict an outcome that then occurs (e.g. a student is not predicted to be at risk, but then stops participating partway through the course). In either case, the model can only improve if some investigation is conducted on why the model was not accurate. To investigate these inaccuracies, the following steps should be taken: 1: In the transition from "prediction" to "training" data set, the element (student, course, cohort, etc.) should be checked to see if its prediction was accurate. If not, the prediction should be marked as either False Positive or False Negative. This is especially important for cases with a large "Cooks Distance." 2: False predictions should be reviewed by a relevant user who might be able to identify a reason the prediction was not accurate. In the case of a prediction about the outcome of a student enrollment in a course, this could be the teacher, the student, or both. See 3: To make analysis simple, probable causes for model inaccuracy should be presented as a checklist that the user can select from, in addition to providing a free-text response. Examples for a False Positive prediction of a student at risk of dropping out might include: Intervention by the teacher helped the student The prediction notice and advice provided to the student helped to change the outcome One or more of the predictors are considered inappropriate by the reviewer (variables should be presented in order of largest deviance residual to smallest) Circumstances unrelated to the course changed the outcome (e.g. the student suddenly had a lot more or less time to work on the course than before) Ethically, it is critical at this point to reveal to any users about whom predictions have been made what those predictions were, on what basis the predictions were made, and what information related to the predictions the system will continue to store and possibly use in future predictions. 4: Summary reports for each model categorizing failures should be available to the system/analytics administrator. Note that it is also possible to conduct forensics on training data that do not fit the model. Care should be taken to avoid over-fitting the model, but examination of false positives and negatives may help to identify novel indicators that should be incorporated into the model.
9.summary: Access to learning analytics models should be role-based
description: Currently the full details of learning analytics models are only available to managers and site administrators, and teachers do not have the option to create or enable models for their own classes. It should be possible for teachers (and eventually, even students) to select from available models, review accuracy calculations, and copy, edit, and create models for their own contexts. Anyone whose data is included in a model or about whom a model makes predictions should have the ability to review the accuracy of the model by default. This may also include users who have roles with respect to other users, e.g. mentors or parents. Teachers may wish to try variant models (e.g. removing or adding some indicators) and comparing them to the standard models used across the site. This can be especially helpful for teachers who teach multiple instances of a single course, either concurrently or over time, or who teach very large enrollment courses. Since processing site-wide models can be computationally expensive, creating a model that uses data outside the contexts visible to a user should generate a request for processing that can be approved by a manager or administrator with appropriate permissions.
10.summary: Report how outcomes change as a result of learning analytics insights and notifications
description: proposes a set of standard calculations of model accuracy. However, we are not concerned only with model accuracy. In fact, we hope for models to be inaccurate in specific ways. A "risk" model (predicting an undesirable event, such as a student dropping out of a course) will ideally have a high false positive rate as a result of changes made by the teacher and student based on insights of the learning analytics model (though a high false positive rate in itself is not what we want). Similarly, a "positive outcome" model (predicting a desirable event, such as a student achieving all competencies attached to the course) will hopefully have a high false negative rate. For this reporting to be facilitated, models need to record whether they are predicting a positive or negative outcome. Changes in predictions over time need to be visible at the individual prediction and overall model analysis level. Viewing rates of insights need to be tracked per model and per role type Actions taken per insight and per role type need to be tracked Changes in outcomes based on these insights and actions need to be monitored. Simplest version: in list of detailed information about models, show whether/how final targets changed from target predictions during analysis period.
11.summary: Provide tools to manage and edit insights
description: Currently, insights (notifications and action opportunities) for learning analytics are programmed in PHP. It should be possible for a user with appropriate permissions to create and edit these in a form. This should allow a view of the model with attached insights, and editing or creating an insight should provide the following controls: Insight name Insight recipient (roles to be checked in the context of the sample) Target sensitivity (e.g. >= 75) Delivery limit - how many times this insight will be delivered to the same recipient for this sample. Default = 3. Insight text with tokens similar to those used in Event notifications Action checkboxes for actions to provide to recipient (tokens Positive actions Send message to \sample (applicable if model makes predictions about a user) View messages with \sample View prediction details Acknowledged Negative actions Not useful Disable insights to me for this \sample (student, course, etc.) Disable insiglts to me for all \samples in this \analysable Link to reports - list of potential reports based on model context. Allows multiple selection. Positive and negative actions help to measure usefulness of model and impact of actions. Prediction follow-up functionality has been moved to .
12.summary: Provide service to collect and distribute learning analytics models and training data sets
description: It has been very difficult to gather full data sets from institutions with which to develop, train, and test learning analytics models. We hope that more models will be developed in partnership with institutions, or independently by institutions using Moodle. To help facilitate this, we could provide a service that automates sharing models and model training data sets. This service might become part of MoodleNet, but would require work within Moodle to facilitate ease of use. Moodle would need the following features in Core: Opt-in trigger in site registration to submit data on what models are in use on the system, their accuracy, and their usage rate (i.e. how many users are receiving insights and what proportion of those insights are being viewed and/or acted on). Some metadata about the site should also be included, i.e. from . Note that some of this data is not currently collected within Moodle sites, but there are proposals to do so, e.g. , . Administration tool to submit a model definition in a reusable form (e.g. PMML) to the Moodle-supplied service - see Administration tool to submit non-identifying model data in csv form to the Moodle-supplied service, identified by a hash code or other mechanisim to tie the data to the model used to generate it - this exists as a manual export capability now, but submission needs to be very easy (even with an automatic option) to encourage sites to submit. Administration tool to select, download and import a model definition, as generated in 2, based on information provided about the model accuracy, testing status, and usage (number of sites, number of users, type of site, as collected in 1) Administration tool to select, download and import model training data, as generated in 3, based on information provided about the accuracy of the model when trained on that data set across multiple sites, types of sites, etc. Im sure this proposal needs to be refined further.
13.summary: Include learning analytics usage data in site registration
description: In order to improve our learning analytics tools to meet community needs, we need to know how many sites are using the existing features and models. This request is to include an opt-in trigger in site registration to submit data on what models are in use on the system, their accuracy, and their usage rate (i.e. how many users are receiving insights and what proportion of those insights are being viewed and/or acted on). Some metadata about the site should also be included, i.e. from . Note that some of this data is not currently collected within Moodle sites, but there are proposals to do so, e.g. , . The following specific information might be included in the site registration statistics: Number of models _installed_ on site Number of models _trained_ on site Total number of analysables used to train models Total number of samples used to train models Minimum, median, and maximum accuracy calculation of _trained_ models Number of models _enabled_ on site Minimum, median, and maximum accuracy calculation of _enabled_ models Total number of analysables processed in most recent model execution Total number of samples processed in most recent model execution Total number of insights generated/sent in most recent model execution Total number of insights viewed since most recent model execution Total number of insight actions taken since most recent model execution Total number of users with analytics:listinsights capability Site education level ISCED code (from ) Default percentage of student work conducted online in Moodle (from )

Below is an instruction that describes a task, paired with an input that provides an epic with this format: 
"project": "Name of the project" 
"summary": "Summary of the epic user story" 
"description": "Description of the epic user story". Write a response that appropriately completes the request.### Instruction:
Break this epic user story into smaller tasks to help the developers understand the epic better, you can divide this epic into as much as you want depending on the complexity of the epic.
### Input:
project: Moodle
summary: Project Inspire Phase II proposal
description: This Epic holds the proposed plan for Project Inspire Phase II, in which the community will be empowered to create, test, and share models, without requiring PHP coding or Moodle developer skills. The following issues are of the highest priority: All bugs identified in 3.4 and 3.5 testing cycles Metadata additions: Site , Activity and Course (necessary to improve model fits and generalizability) Allow targets to override access control to predictions (otherwise we cannot deliver insights to anyone except instructors and managers) Allow model configurations to be imported and exported (development already in progress) Implement UI for creating and editing learning analytics based on reports - this is the key new feature, which will make it possible for non-developers to construct models and share them, including model training data. This will lower the priority of many of the other feature requests in the backlog. Alternatively, is a proposal to implement a UI for creating and editing simple learning analytics models based only on events. For the use of analytics in large courses, especially MOOCs, the following is critical: Allow bulk actions to notifications from learning analytics Finally, Display accuracy of models more prominently and with more detail is an important feature to support researchers developing and testing models.

### Response:
1.summary: Display accuracy of models more prominently and with more detail
description: Currently the accuracy of models is reported only in the Log action menu. Model accuracy is a key aspect of learning analytics, and needs to be displayed more prominently. Accuracy varies with time-splitting methods, but only one time-splitting method can be active for a model at a time, so the accuracy level for the currently used time-splitting method should be displayed on the list of models. More detail should be available about the accuracy of each model, as well. One simple way to display this information is by using a ROC curve plotting the true positive rate of the model predictions against the false positive rate: Display the following details for each model: Contingency distribution (graph) ROC curve (graph) Area under curve (number) Contingency table and diagnostics: (note that most of these are simple calculations of a small number of basic values) True positive n Power Recall/Sensitivity (TPR) False positive n Type I error Fall-out (FPR) False negative n Type II error Miss rate (FNR) True negative n Specificity (TNR) Predicted positive n Prevalence Precision (PPV) LR+ Predicted negative n False omission (FOR) LR- Condition positive n Condition negative n Total n Diagnostics: Accuracy False discovery rate Negative prediction value DIagnostic odds ratio F1 score Detail of indicators: Selected indicator ROC curves (graph) with control to select how many indicators to display Model effect size (standardized) Indicator graphs: Indicator significance (p-value) Standardized effect sizes with confidence intervals for all indicators _Alternative charts, such as the detection error tradeoff graph or zROC could be used as the header graph instead of a ROC. Usability studies should determine which kind of display would be easiest for most users to understand and use in decision-making._ _Not that the ROC curve is only applicable for binary predictions, but at present these are the only prediction models supported by Moodle learning analytics. This could be extended to REC or RROC curves once we extend support for other types of predictions._
2.summary: Enhance enrolment record to include "expected start" and "expected end" dates
description: In "rolling enrollment" courses, there may be no single course start date. Because of this, the due dates for activities could be different for each student. suggests that the due dates for activities be allowed to be set as deltas from the student enrollment date, but the date the student was enrolled in the course may not be their expected start date. It could be beneficial to modify the enrollment record so that it contains both an enrolled date and an "expected start" date. Activity deadlines would be set based on the "expected start" date, rather than the enrollment date. Similarly, the enrollment record could contain two ending dates, one switching the enrollee to read-only access, the other preventing any access to the course. Any "end" dates for enrollment could be calculated based on the "expected start" date. It is also possible that one might want to set due dates (and course enrollment expected start dates) relative to the enrollment end date. This is key to supporting learning analytics for rolling-enrollment courses.
3.summary: Improve Predictions report
description: The current report of insights generated by learning analytics is sparse (see attached image) and could be more immediately informative. Currently the report consists of sections per prediction, and each section contains two columns: Name (or other sample entity) Actions menu Please add: likelihood of prediction Selection checkbox Bulk action function per prediction type Under Actions, please add "Review messages" as a way to quickly jump to the message history with the user the prediction is about (if it is about a user).
4.summary: Improve prediction details report
description: The current prediction details report generated by learning analytics could be more informative. Currently the report consists of a header with the name of the prediction, the name of the user (or other sample entity) the prediction is about, a menu of actions, and the date of the prediction. The date range covered by the prediction is in a long text form which is difficult to read. (See attached screenshot.) Below the header, indicators and calculated values are displayed in two columns. The rows are color-coded by the calculated values of each indicator. The spacing of the rows is loose, so it is difficult to get an overview of the status of all indicators at once. Please add to header: likelihood of prediction Under Actions, please add "Review messages" as a way to quickly jump to the message history with the user the prediction is about (if it is about a user). For each indicator, please add: A link to help text for that indicator Weight value in the model for that indicator Number of times that indicator has occurred for this sample (if it is an event)
5.summary: Improve analytics models display and administration
description: This is a proposal to make the management and display of the list of analytics models easier to read and more informative. In this proposal, the current list of analytics models would be slightly simplified (removing the indicator display and truncating the width of the "insights" drop-down), and the ability to add, duplicate, edit, and delete models using one of several potential model editors (e.g. or ) would be implemented. It would be possible to export, share, and import models, perhaps as xml or JSON files, including training data. This display would also highlight model training status and accuracy and usage calculations, to aid in evaluation of models.
6.summary: Invalid site elements report needs summary
description: The "Invalid site elements" report can go to multiple pages, but does not give any hint as to how many pages there will be. There is also no way to quickly determine the reasons for the invalid elements. The following enhancements are recommended: As with all analytics pages, please ensure that breadcrumbs are displayed and work. Display pager that indicates how many pages in total are in the report Provide summary on first page listing: How many courses are invalid to train Proportional breakdown of reasons those courses are invalid How many courses are invalid to get predictions Proportional breakdown of reasons those courses are invalid Other enhancements that would be helpful include display of additional information, e.g. categories, start/end dates, visibility, filters to view only courses which satisfy criteria (e.g. reason for invalidity), dates, categories, etc., ability to adjust number of elements per page of report, ability to sort by different criteria, etc. For reasons which could be corrected, e.g. course does not have an end time, provide link to a bulk editing tool to adjust end times, etc. (see related issue ) Allow elements to be "acknowledged" so they can be filtered from the report, so only new invalid elements can be displayed.
7.summary: Provide ability to edit course metadata in bulk
description: Several metadata fields attached to courses affect learning analytics and other reports, and some of these have been recently introduced or may have been used inconsistently on a site in the past. This feature would allow the site administrator or manager to filter courses by criteria (e.g. category, substrings of fullname or shortname, metadata that was filled e.g. startdate) and select courses for bulk update of any universal course fields/settings, including: Start date End date Visibility Add/Remove competency Course format Gradebook visibility Completion tracking Group mode Tags All metadata referenced in (Add Course metadata to support learning analytics) Instructional mode (online vs. hybrid vs. F2F) online activity Absolute/relative dates Expected time to complete Course level Subject area
8.summary: Provide tools to conduct forensics on analytics when prediction fails
description: The learning analytics system makes predictions about educational outcomes such as student success or course effectiveness. Ideally, the analytics model will be highly accurate in its predictions. However, two kinds of errors should be reported and investigated: Type I or "False positive," when the model predicts that an outcome will occur but it fails to occur (e.g. a student is predicted to be at risk of dropping out, but the student continues in the course until completion) and Type II or "False negative," when the model fails to predict an outcome that then occurs (e.g. a student is not predicted to be at risk, but then stops participating partway through the course). In either case, the model can only improve if some investigation is conducted on why the model was not accurate. To investigate these inaccuracies, the following steps should be taken: 1: In the transition from "prediction" to "training" data set, the element (student, course, cohort, etc.) should be checked to see if its prediction was accurate. If not, the prediction should be marked as either False Positive or False Negative. This is especially important for cases with a large "Cooks Distance." 2: False predictions should be reviewed by a relevant user who might be able to identify a reason the prediction was not accurate. In the case of a prediction about the outcome of a student enrollment in a course, this could be the teacher, the student, or both. See 3: To make analysis simple, probable causes for model inaccuracy should be presented as a checklist that the user can select from, in addition to providing a free-text response. Examples for a False Positive prediction of a student at risk of dropping out might include: Intervention by the teacher helped the student The prediction notice and advice provided to the student helped to change the outcome One or more of the predictors are considered inappropriate by the reviewer (variables should be presented in order of largest deviance residual to smallest) Circumstances unrelated to the course changed the outcome (e.g. the student suddenly had a lot more or less time to work on the course than before) Ethically, it is critical at this point to reveal to any users about whom predictions have been made what those predictions were, on what basis the predictions were made, and what information related to the predictions the system will continue to store and possibly use in future predictions. 4: Summary reports for each model categorizing failures should be available to the system/analytics administrator. Note that it is also possible to conduct forensics on training data that do not fit the model. Care should be taken to avoid over-fitting the model, but examination of false positives and negatives may help to identify novel indicators that should be incorporated into the model.
9.summary: Access to learning analytics models should be role-based
description: Currently the full details of learning analytics models are only available to managers and site administrators, and teachers do not have the option to create or enable models for their own classes. It should be possible for teachers (and eventually, even students) to select from available models, review accuracy calculations, and copy, edit, and create models for their own contexts. Anyone whose data is included in a model or about whom a model makes predictions should have the ability to review the accuracy of the model by default. This may also include users who have roles with respect to other users, e.g. mentors or parents. Teachers may wish to try variant models (e.g. removing or adding some indicators) and comparing them to the standard models used across the site. This can be especially helpful for teachers who teach multiple instances of a single course, either concurrently or over time, or who teach very large enrollment courses. Since processing site-wide models can be computationally expensive, creating a model that uses data outside the contexts visible to a user should generate a request for processing that can be approved by a manager or administrator with appropriate permissions.
10.summary: Report how outcomes change as a result of learning analytics insights and notifications
description: proposes a set of standard calculations of model accuracy. However, we are not concerned only with model accuracy. In fact, we hope for models to be inaccurate in specific ways. A "risk" model (predicting an undesirable event, such as a student dropping out of a course) will ideally have a high false positive rate as a result of changes made by the teacher and student based on insights of the learning analytics model (though a high false positive rate in itself is not what we want). Similarly, a "positive outcome" model (predicting a desirable event, such as a student achieving all competencies attached to the course) will hopefully have a high false negative rate. For this reporting to be facilitated, models need to record whether they are predicting a positive or negative outcome. Changes in predictions over time need to be visible at the individual prediction and overall model analysis level. Viewing rates of insights need to be tracked per model and per role type Actions taken per insight and per role type need to be tracked Changes in outcomes based on these insights and actions need to be monitored. Simplest version: in list of detailed information about models, show whether/how final targets changed from target predictions during analysis period.
11.summary: Provide tools to manage and edit insights
description: Currently, insights (notifications and action opportunities) for learning analytics are programmed in PHP. It should be possible for a user with appropriate permissions to create and edit these in a form. This should allow a view of the model with attached insights, and editing or creating an insight should provide the following controls: Insight name Insight recipient (roles to be checked in the context of the sample) Target sensitivity (e.g. >= 75) Delivery limit - how many times this insight will be delivered to the same recipient for this sample. Default = 3. Insight text with tokens similar to those used in Event notifications Action checkboxes for actions to provide to recipient (tokens Positive actions Send message to \sample (applicable if model makes predictions about a user) View messages with \sample View prediction details Acknowledged Negative actions Not useful Disable insights to me for this \sample (student, course, etc.) Disable insiglts to me for all \samples in this \analysable Link to reports - list of potential reports based on model context. Allows multiple selection. Positive and negative actions help to measure usefulness of model and impact of actions. Prediction follow-up functionality has been moved to .
12.summary: Provide service to collect and distribute learning analytics models and training data sets
description: It has been very difficult to gather full data sets from institutions with which to develop, train, and test learning analytics models. We hope that more models will be developed in partnership with institutions, or independently by institutions using Moodle. To help facilitate this, we could provide a service that automates sharing models and model training data sets. This service might become part of MoodleNet, but would require work within Moodle to facilitate ease of use. Moodle would need the following features in Core: Opt-in trigger in site registration to submit data on what models are in use on the system, their accuracy, and their usage rate (i.e. how many users are receiving insights and what proportion of those insights are being viewed and/or acted on). Some metadata about the site should also be included, i.e. from . Note that some of this data is not currently collected within Moodle sites, but there are proposals to do so, e.g. , . Administration tool to submit a model definition in a reusable form (e.g. PMML) to the Moodle-supplied service - see Administration tool to submit non-identifying model data in csv form to the Moodle-supplied service, identified by a hash code or other mechanisim to tie the data to the model used to generate it - this exists as a manual export capability now, but submission needs to be very easy (even with an automatic option) to encourage sites to submit. Administration tool to select, download and import a model definition, as generated in 2, based on information provided about the model accuracy, testing status, and usage (number of sites, number of users, type of site, as collected in 1) Administration tool to select, download and import model training data, as generated in 3, based on information provided about the accuracy of the model when trained on that data set across multiple sites, types of sites, etc. Im sure this proposal needs to be refined further.
13.summary: Include learning analytics usage data in site registration
description: In order to improve our learning analytics tools to meet community needs, we need to know how many sites are using the existing features and models. This request is to include an opt-in trigger in site registration to submit data on what models are in use on the system, their accuracy, and their usage rate (i.e. how many users are receiving insights and what proportion of those insights are being viewed and/or acted on). Some metadata about the site should also be included, i.e. from . Note that some of this data is not currently collected within Moodle sites, but there are proposals to do so, e.g. , . The following specific information might be included in the site registration statistics: Number of models _installed_ on site Number of models _trained_ on site Total number of analysables used to train models Total number of samples used to train models Minimum, median, and maximum accuracy calculation of _trained_ models Number of models _enabled_ on site Minimum, median, and maximum accuracy calculation of _enabled_ models Total number of analysables processed in most recent model execution Total number of samples processed in most recent model execution Total number of insights generated/sent in most recent model execution Total number of insights viewed since most recent model execution Total number of insight actions taken since most recent model execution Total number of users with analytics:listinsights capability Site education level ISCED code (from ) Default percentage of student work conducted online in Moodle (from )
