by Bobby Siegfried.
Got it working! Here's what I did last. Thank you for your suggestion of going back to the mdl_backup_courses table. That ended up being the right place. Alian's post was really helpful as well. I'm thinking of writing up some documentation for Moodle on this subject. The automated backup task feels like one of the most important tasks in the system, but the process is quite the mystery I'm finding! Below I'll share everything I did, even though it will recap some of our previous posts in hopes it helps someone else down the line.
So in full, when I first encountered the issue, one of the first things I did was run the automated backup cleanup task. It had run with the cron run, but I ran it again manually just to be extra sure. It completed successfully both times but that didn't resolve the issue.
Upon your suggestion, I took a look at moodledata/temp/backup/ to see if there were any unfinished backups (hashed directories that were not 0 byte .log files) and found 3. I removed those three directories. This did not resolve the issue.
Then I took a look in the mdl_backup_controllers table because I had seen some posts where it seemed like this table was more than just a reporting table - however as you said it does appear to be simply a reporting table. Nevertheless, I found some helpful information in the Moodle backup class code that actually describes the "status" codes well (Note we are running 3.7.x at this time so that's what this code is). I removed the 3 entries for the failed courses (had a status of 800 which is "currently executing") in case that was causing the backup task to fail - again not the case.
So in full, when I first encountered the issue, one of the first things I did was run the automated backup cleanup task. It had run with the cron run, but I ran it again manually just to be extra sure. It completed successfully both times but that didn't resolve the issue.
Upon your suggestion, I took a look at moodledata/temp/backup/ to see if there were any unfinished backups (hashed directories that were not 0 byte .log files) and found 3. I removed those three directories. This did not resolve the issue.
Then I took a look in the mdl_backup_controllers table because I had seen some posts where it seemed like this table was more than just a reporting table - however as you said it does appear to be simply a reporting table. Nevertheless, I found some helpful information in the Moodle backup class code that actually describes the "status" codes well (Note we are running 3.7.x at this time so that's what this code is). I removed the 3 entries for the failed courses (had a status of 800 which is "currently executing") in case that was causing the backup task to fail - again not the case.
Finally, upon your suggestion and Alian's post that you shared, I located the entries in the mdl_backup_courses table for each of the 3 problem courses and found only two of them had "laststatus" set to 0, which I learned from another post of yours, Ken, means that the backup is "unfinished". More details on those status codes I found in your post and in the Moodle code: https://github.com/moodle/moodle/blob/MOODLE_37_STABLE/backup/util/helper/backup_cron_helper.class.php. I changed the status of both to 1 (status OK), but I did not change the "nextstarttime" at that point because I wanted to see what simply changing the laststatus value did. After updating the "laststatus" value, the automated backup task ran and completed successfully. Interestingly enough, the problem courses both reported a status of Skipping due to neither course being modified in the past 30 days (which is expected since these courses are both over a year old and not actively changing). After the first run of the automated backup task, the nextstarttime value was set to Saturday at 3:50am which is when the next scheduled run of automated backups is set for. I ran the task manually one more time and it completed successfully again, no more "error_setting_already_set". So it looks like the table mdl_backup_courses is the one to look at and potentially make changes to.
I'll update again after the task runs at its scheduled time on Saturday beginning at 3:50am and see if it runs into any further issues. My theory is that everything should be good at this point. Statuses in the related db tables are all good and the temp directories of the failed backup attempts have been removed.
Thank you Ken for some great insight here, your assistance has been tremendous.