I guess am not explaining well enough ...
"Auto Backup Starts > backup one course > store in temp/backup > compile > copy to NFS"
Terminology aside, yes, that is what happens with one addition ... after backup has been copied to destination, the actual routine is to erase the temp directory used to build the backup.
Catch 22 ... a build of a backup.mbz file fails or *copy* of backup.mbz fails - for whatever reason ... since autobackup code doesn't get notice that course backup has failed - database tables related don't get a completed ... nothing removed. However there is a clean up task (part of moodle cron) that is supposed to clean up.
Depending upon version of Moodle, an adhoc_task.php script attempts to pick up where things left off - no completion notice.
"Is there a way to delete the temp/backup's copy in real time right after it has been copied to NFS?"
See above!
If you do any other/outside of Moodle cron/task jobs to remove whatever is in moodledata/backup/temp/ I don't know how you can do that without disrupting an autobackup during a build. If anything, you should be running the clean up task ... not a home grown cron to clean up.
How long each of those 1000's of courses takes depends upon each course. I've never seen any documentation that says if a course is X Gig in size it will take Y minutes to backup and copy to destination.
With 1000's of courses I doubt autobackups complete in one day.
Since you have done designated directories, can you determine the largest backup file in that designated directory? Actually verify that what you say "256G in backups is only 1 day's worth" is true? Both in meta data from DB and physically in designated directory?
Now while all this is going on ... are your teachers deleting modules in their courses? If so, those deletions create recyclebin backups which are on a schedule of their own when it comes to keeping, moving to trash, and then eventually removal. Plus, those course backups made by auto can't be accurate any more and might actually ballon a backup if there are recyclebin backups associated with the course.
Also remember, if auto takes all day or 3 days ... what of the 'normal' daily traffic to site? Ever max out memory of code server?
'SoS', Ken