Thanks for the additional information. Can now be a little more specific as to what to 'watch'.
Might need to increase the php settings for exec to something larger than 60 seconds. Those Gig+ courses take time to build. That could be the reason large courses fail to complete. BTW, how would Moodle know to cleanup a directory IF the script creating the backup fails? Answer ... don't think it would.
So when I run into one of those, I'll first clean the /moodledata/backup/temp/ folder of anything that resides there ... .log files, temp hashnamed directories and hashednamed files.
Then, from command line, while in /moodledata/backup/temp/, issue the following command:
watch "ls -lR" [ENTER]
You can watch that window and see Moodle build the backup ... can also see where it stops.
In another terminal session, from the command line, I'll use the following:
tail -f /var/log/httpd/error_log
On Ubuntu think that's tail -f /var/log/apache2/error.log (or something like that)
That allows me to watch that log in realtime.
And yet, in another, run 'top' ... that will show php kicking in from time to time ... and how much memory it's using, etc..
In the browser, I'll go to that course and begin a backup process on one large course that failed. If you can run the browser backup on from another workstation it's easier to see what is happening/or not on the workstation you are using for terminal access. Or, in yet another terminal session, run the automated backup script from the command line. Yeah, I know three terminal screens. :\
Have discovered different issues (doing the above) so can't say for sure what you'll see/find.
Basically, moodle creates a 'roadmap' of what to backup in moodle_backup.xml so if the script dies before it can write info about files to that .xml file, the files.xml file and then copy the contenthashed named file from moodledata/filedir/whereever ... one might be able to determine which file.
Will also have to do a query of the DB to see if teachers are making their own backups and how many they have. Moodle creates those backups with date/time stamps in the names so they are not the same files each time a teacher backs up the course. You could have multiple backups IN the backup you are attempting to create.
So this query should get some info for ya:
select contenthash,filename,filesize from `mdl_files` where `filename` like "%.mbz%"
If the mount point for moodledata/backup/temp is on another box, that will make the build process slower (which means an increase in the time to exec) but it's off the web server.
'spirit of thinkering', Ken