by Ken Task.
Does automated hit the same course and fail? That's one thing I've always hated about autobackups ... no way to exclude a course or courses having issues. Auto hits the course having issues, then doesn't recover well ... stops. Builds up each night ... until????
You might have to resort to finding out which courses the largest and then creating a bash shell script to handle just those in one run and then another bash shell script to handle all other others.
When pending adhoc builds up you might never catch up.
When it gets that bad, one might need to truncate the tables that track auto. Moodle has no record of anytthing auto then thus it would be like the first time it ran ... but ... you hit those same courses having issues, and back to square one.
Inspection of failed backup directories in moodledata/temp/backups/ might give some clues ... minimally the course ID's having issues and thus could exclude those ID's in the bash shell script that loops through courses and runs moosh to back them up.
18140 courses .. skipped 16014 ... so what's the status of those that were skipped? Are they still being used? There is an option in auto to skip courses that have not changed.
Your situation might be due to the diff between 3.highest and 4.highest ... always major changes in such upgrades, but wonder, if you have been facing this same thing, year after year, if an overall change in strategy might need to be considered.
There are entities that clone the site to another server - different fully qualfied domain name - like lms2023-24.tld - get that running, upgrade, then use moosh to reset all courses ... backup preferences changed to make no user backups. Run one set of course backups via looping clii script that saves those nu's to an archive directory. After that completes, change back backup preferences for when teachers might use.
Auto backups not turned on until after the start of the next academic session/semester ... say 2 weeks into.
Any course not to be used is deleted ... recyclebin set to keep for 1 second.
Now while all that is going on, a MySQL Tuner check is run ... tweaks applied as needed. Ditto PHP settings ... time for a script can run, memory a script can consume ... many things boosted upwards to handle the massive work being done. After completion of prep, those might need to be reset back to original settings for remainder of academic session.
Since you are not alone in admin-ing your Moodle, might want to talk to your IT/server/db admins, etc. persons.
'SoS', Ken