The error happened again last night, at about the same time. I've attached a section of the cron log between 00:49 and 1:00 when all the courses have finished backing up.
Here is the relevant snippet:
Server Time: Thu, 03 Sep 2015 00:57:01 -0400
Execute scheduled task: Delete stale temp files
... started 00:57:02. Current memory use 18.4MB.
backup_auto_failed_on_course DemonChem
Exception: directory_not_exists /var/www/vhosts/rjreynolds.org/moodledata/temp/backup/25d3b4b9248732c758fca0aa6588b48c/activities/quiz_7194
complete - next execution: Fri, 04 Sep 2015 00:30:00 -0400
Backing up e-Commerce I (Blair)...
... used 0 dbqueries
... used 2.2312309741974 seconds
Scheduled task complete: Delete stale temp files
Execute scheduled task: Cleanup files in trash
... started 00:57:04. Current memory use 19.5MB.
Deleting old draft files... ... started 00:57:04. Current memory use 21.8MB.
done.
Deleting orphaned preview files... ... started 00:57:06. Current memory use 21.9MB.
done.
... used 11 dbqueries
... used 2.2669699192047 seconds
Scheduled task complete: Cleanup files in trash
I must admit that these logs are hard to look at because different parts seemed to be mashed together sometimes based on when the various tasks complete.
There are also a few of these errors in the log at various points. They start happening during the time that the automated backups are running and don't seem to happen outside that window. Here is that snippet:
Default exception handler: The operation timed out while waiting for a lock. Debug:
Error code: locktimeout
* line 446 of /lib/classes/task/manager.php: moodle_exception thrown
* line 66 of /lib/cronlib.php: call to core\task\manager::get_next_scheduled_task()
* line 61 of /admin/cli/cron.php: call to cron_run()
!!! The operation timed out while waiting for a lock. !!!
I'm now in officially over my head as far as interpreting these things so I'll appreciate whatever you have for me. If it would be helpful to you, I do have the log saved from last night when all backups completed successfully as well as a much longer log file that I pulled this piece out of.
Thanks all.