project: Moodle
summary: META: Logging stage 2
description: This project is a major overhaul of the current logging system in Moodle which has remained almost unchanged since Moodle 1.0. Full specifications are here:

1.summary: Define logging interfaces in core
description: - Define logging interfaces in core - Define the expected log table structure with extra request information Interfaces are listed in :
2.summary: Convert Activity report (outline) to use new logging API
description: Report should have a setting which log SQL reader plugin to use (i.e. log_standard, log_legacy)
3.summary: Convert recent activity block to use new Events API
description: 
4.summary: Investigating performance of the new logging API in practice
description: Once the new logging API is in place, its performance and impact need to be measured. Actions to reduce the impact of the new logging system should be investigated.
5.summary: Replace old add_to_log function calls to use event triggers
description: The add_to_log function needs to be replaced with the new event triggers. The event trigger will contain legacy data to replicate the add_to_log function for backwards compatibility. This will need to be removed as a sub-task and made a meta issue to change the 329 calls to add_to_log.
6.summary: Finish detailed specification
description: First task for the BACKEND team is to finish a detailed technical specification of the new system here and get approval from stakeholders:
7.summary: Add support for admin tool subplugins
description: This was requested a few times already, since 2.6 this does not have any significant performance cost. It could help other tools too. The only potential problem is that admin tools are in $CFG->dirroot/$CFG->admin/tool, we need to deal with alternative $CFG->admin !== admin
8.summary: Improve performance of log table
description: There is somewhat of a performance bug in the way logs/activity reports are handled. Activity reports are of course generally slow, and tend to run some very SQL selects. While these are selects are happening, of course no inserts can occur. This blocks essentially all traffic on the site (since almost any page view/action includes a log write). This directly relates to the count of log rows you have. In our case we pruned it as much as we could, and were left with 10M+ entries (we have to keep at least 2 semesters, worth of logs), so each select may take 5-15s, which really cripples the site. The fix that we implemented is to have 2 tables. One,log, and one log_insert. We modified the add_to_log in datalib.php to write to log_insert instead of log. Every cron cycle a piece of code run that moves log entries from log_insert to log. Martin D. mentioned that logs were one of the things he isnt very happy with the general implementation of, so maybe there needs to be a bigger discussion of it. Ill post our patch code when I get back to the states, but its pretty basic.
9.summary: Implement new log storage management admin UI
description: Implement new log storage management UI Note that this issue is not blocked by anything, it is just one page with list of stores with some toggles and links to individual admin setting pages
10.summary: Reproduce log reports to use the new logging system (log and loglive)
description: Convert reports /report/log/ and /report/loglive/ to the new API Reports must work nicely with both log_legacy and log_standard Alter reports to accept new event object instances instead of array data. Alter reports to understand new event properties. For each report add a setting which log reader to use Deprecate function print_log(), move html to corresponding renderers Deprecate functions print_log_ods(), print_log_csv(), print_log_xls() - they should all be part of /report/log/ Deprecate function build_logs_array() Deprecate function get_logs_usercourse(), move code into /report/log/
11.summary: Buffering in standard log storage
description: Improvement to Moodle standard DB log storage supporting buffering and performance tuning
12.summary: Convert Course participation report to new logging API
description: 
13.summary: Write transition guide for reports that rely on logs
description: A docs page should be written for developers to describe how to replace their current direct log queries with logging readers.
14.summary: Write transition guide for events and logging
description: A docs page for developers is needed to describe how to replace the current use of add_to_log() with event triggers.
15.summary: Remove access to old log table in admin/handlevirus.php and other clam related code
description: Currently this library uses the legacy logging table. It needs to be upgraded to use the new logging API.
16.summary: Reimplement access to log table in backup/util/helper/backup_cron_helper.class.php
description: Currently this class uses the legacy logging table. It needs to be upgraded to use the new logging API.
17.summary: Reimplement access to log table in function can_delete_course()
description: Currently this function uses the legacy logging table. It needs to be upgraded to use the new logging API.
18.summary: Reimplement access to log table in functions notify_login_failures() and count_login_failures()
description: Currently these functions uses the legacy logging table. It needs to be upgraded to use the new logging API.
19.summary: Deprecate $event->can_view() until its purpose has been clarified
description: The can_view() method in event base looked like a good idea, unfortunately it is not easy and cheap to implement it everywhere. At preset it is returns true for admin only which makes it unusable in reports. At present the reports are expected to use own capability together with \core\log\reader::can_access($reportcontext) and ignore the \core\event\base::can_view() method.
20.summary: Convert Statistics report to new logging API
description: The idea is to do this with minimal effort faking only the contents of the temp tables based on new crud attribute...
21.summary: Convert outline report to use new logging API
description: The outline report itself does one query in log but also each module implementing _outline() does it too, see Instead the function returning the entries from log should be inside the outline report and each module should call this function. Outline report will have a setting which log storage to use.
22.summary: Implement an external logging plugin
description: To exemplify and justify the new logging system, it is essential to implement a logging plugin with writer and reading support, to a system external to Moodle. This will also act as a template/example for contributed logging plugins.
23.summary: Implement log buffering to delay writes to the end of the process/page load
description: If there are a number of events triggered in a page, the processing of these events, particularly in relation to logging, may be more efficiently processed in a single chunk, hopefully reducing the number of database writes.
24.summary: Reproduce log reports to use the new logging system (log)
description: Reproduce log reports to use the new logging system (log)
25.summary: Rewrite loglive report to use the new logging api
description: Reproduce log reports to use the new logging system (loglive)
26.summary: Switch to the new Standard log store by default
description: Sites upgraded to 2.7 should be adding new logs to the new standard logging table only, however we need to keep legacy logging enabled for reading only so that the non-core reports can still show older data. We will encourage people upgrading to disable the legacy log over time. In newly installed sites legacy logs should be disabled completely by default.

Below is an instruction that describes a task, paired with an input that provides an epic with this format: 
"project": "Name of the project" 
"summary": "Summary of the epic user story" 
"description": "Description of the epic user story". Write a response that appropriately completes the request.### Instruction:
Break this epic user story into smaller tasks to help the developers understand the epic better, you can divide this epic into as much as you want depending on the complexity of the epic.
### Input:
project: Moodle
summary: META: Logging stage 2
description: This project is a major overhaul of the current logging system in Moodle which has remained almost unchanged since Moodle 1.0. Full specifications are here:

### Response:
1.summary: Define logging interfaces in core
description: - Define logging interfaces in core - Define the expected log table structure with extra request information Interfaces are listed in :
2.summary: Convert Activity report (outline) to use new logging API
description: Report should have a setting which log SQL reader plugin to use (i.e. log_standard, log_legacy)
3.summary: Convert recent activity block to use new Events API
description: 
4.summary: Investigating performance of the new logging API in practice
description: Once the new logging API is in place, its performance and impact need to be measured. Actions to reduce the impact of the new logging system should be investigated.
5.summary: Replace old add_to_log function calls to use event triggers
description: The add_to_log function needs to be replaced with the new event triggers. The event trigger will contain legacy data to replicate the add_to_log function for backwards compatibility. This will need to be removed as a sub-task and made a meta issue to change the 329 calls to add_to_log.
6.summary: Finish detailed specification
description: First task for the BACKEND team is to finish a detailed technical specification of the new system here and get approval from stakeholders:
7.summary: Add support for admin tool subplugins
description: This was requested a few times already, since 2.6 this does not have any significant performance cost. It could help other tools too. The only potential problem is that admin tools are in $CFG->dirroot/$CFG->admin/tool, we need to deal with alternative $CFG->admin !== admin
8.summary: Improve performance of log table
description: There is somewhat of a performance bug in the way logs/activity reports are handled. Activity reports are of course generally slow, and tend to run some very SQL selects. While these are selects are happening, of course no inserts can occur. This blocks essentially all traffic on the site (since almost any page view/action includes a log write). This directly relates to the count of log rows you have. In our case we pruned it as much as we could, and were left with 10M+ entries (we have to keep at least 2 semesters, worth of logs), so each select may take 5-15s, which really cripples the site. The fix that we implemented is to have 2 tables. One,log, and one log_insert. We modified the add_to_log in datalib.php to write to log_insert instead of log. Every cron cycle a piece of code run that moves log entries from log_insert to log. Martin D. mentioned that logs were one of the things he isnt very happy with the general implementation of, so maybe there needs to be a bigger discussion of it. Ill post our patch code when I get back to the states, but its pretty basic.
9.summary: Implement new log storage management admin UI
description: Implement new log storage management UI Note that this issue is not blocked by anything, it is just one page with list of stores with some toggles and links to individual admin setting pages
10.summary: Reproduce log reports to use the new logging system (log and loglive)
description: Convert reports /report/log/ and /report/loglive/ to the new API Reports must work nicely with both log_legacy and log_standard Alter reports to accept new event object instances instead of array data. Alter reports to understand new event properties. For each report add a setting which log reader to use Deprecate function print_log(), move html to corresponding renderers Deprecate functions print_log_ods(), print_log_csv(), print_log_xls() - they should all be part of /report/log/ Deprecate function build_logs_array() Deprecate function get_logs_usercourse(), move code into /report/log/
11.summary: Buffering in standard log storage
description: Improvement to Moodle standard DB log storage supporting buffering and performance tuning
12.summary: Convert Course participation report to new logging API
description: 
13.summary: Write transition guide for reports that rely on logs
description: A docs page should be written for developers to describe how to replace their current direct log queries with logging readers.
14.summary: Write transition guide for events and logging
description: A docs page for developers is needed to describe how to replace the current use of add_to_log() with event triggers.
15.summary: Remove access to old log table in admin/handlevirus.php and other clam related code
description: Currently this library uses the legacy logging table. It needs to be upgraded to use the new logging API.
16.summary: Reimplement access to log table in backup/util/helper/backup_cron_helper.class.php
description: Currently this class uses the legacy logging table. It needs to be upgraded to use the new logging API.
17.summary: Reimplement access to log table in function can_delete_course()
description: Currently this function uses the legacy logging table. It needs to be upgraded to use the new logging API.
18.summary: Reimplement access to log table in functions notify_login_failures() and count_login_failures()
description: Currently these functions uses the legacy logging table. It needs to be upgraded to use the new logging API.
19.summary: Deprecate $event->can_view() until its purpose has been clarified
description: The can_view() method in event base looked like a good idea, unfortunately it is not easy and cheap to implement it everywhere. At preset it is returns true for admin only which makes it unusable in reports. At present the reports are expected to use own capability together with \core\log\reader::can_access($reportcontext) and ignore the \core\event\base::can_view() method.
20.summary: Convert Statistics report to new logging API
description: The idea is to do this with minimal effort faking only the contents of the temp tables based on new crud attribute...
21.summary: Convert outline report to use new logging API
description: The outline report itself does one query in log but also each module implementing _outline() does it too, see Instead the function returning the entries from log should be inside the outline report and each module should call this function. Outline report will have a setting which log storage to use.
22.summary: Implement an external logging plugin
description: To exemplify and justify the new logging system, it is essential to implement a logging plugin with writer and reading support, to a system external to Moodle. This will also act as a template/example for contributed logging plugins.
23.summary: Implement log buffering to delay writes to the end of the process/page load
description: If there are a number of events triggered in a page, the processing of these events, particularly in relation to logging, may be more efficiently processed in a single chunk, hopefully reducing the number of database writes.
24.summary: Reproduce log reports to use the new logging system (log)
description: Reproduce log reports to use the new logging system (log)
25.summary: Rewrite loglive report to use the new logging api
description: Reproduce log reports to use the new logging system (loglive)
26.summary: Switch to the new Standard log store by default
description: Sites upgraded to 2.7 should be adding new logs to the new standard logging table only, however we need to keep legacy logging enabled for reading only so that the non-core reports can still show older data. We will encourage people upgrading to disable the legacy log over time. In newly installed sites legacy logs should be disabled completely by default.
