The one moodle instance primarily used as a quiz machine is heavy processing ... at any time ... students taking or even during backups. So you might look into how those quizzes are setup ... all questions on one page? 5 questions per page? Could be you have some quiz related tables that are growing large. Larger tables = longer processing.
Here is a query I found (didn't create) that shows largest tables:
SELECT CONCAT(table_schema, '.', table_name),
CONCAT(ROUND(table_rows / 1000000, 2), 'M') rows,
CONCAT(ROUND(data_length / ( 1024 * 1024 * 1024 ), 2), 'G') DATA,
CONCAT(ROUND(index_length / ( 1024 * 1024 * 1024 ), 2), 'G') idx,
CONCAT(ROUND(( data_length + index_length ) / ( 1024 * 1024 * 1024 ), 2), 'G') total_size,
ROUND(index_length / data_length, 2) idxfrac
FROM information_schema.TABLES
ORDER BY data_length + index_length DESC
LIMIT 10;
On a K12 site 'smaller' than yours but does a lot of quizzes throughout the site:
| moodle2.mdl_grade_grades_history | 2.31M | 0.21G | 0.34G | 0.55G | 1.66 |
| moodle2.mdl_question_attempts | 0.46M | 0.16G | 0.04G | 0.19G | 0.23 |
| moodle2.mdl_question_attempt_step_data | 1.91M | 0.06G | 0.02G | 0.09G | 0.40 |
| moodle2.mdl_question_attempts | 0.46M | 0.16G | 0.04G | 0.19G | 0.23 |
Also, when was the last time the DB for that instance was checked with some tool like MySQLTuner?
You didn't mention if the DB server is dedicated or does each of your servers 'do it all' ... code + DB + files?
What does the 'top' of 'top' look like?
'spirit of sharing', Ken