Hello,
Will course backup/restore with user-data able to create missing users on the second site?
Yes.
What if courses/ users already exist in a new instance?
As far as I know, if a course already exists, a duplicate course will be created with 'copy 1' appended to the name. However, if users already exist, duplicate users will not be created.
Will backup/restore include submission files (like images)?
Yes, when you backup with user data.
Is there any other better solution for merging data on two different moodle instances?
I think the solution you mention is the best, though before you create the backups, you may want to upgrade your 2.8 site, as a lot has changed between 2.8 and 3.9.1!
What precautions need to be taken before/while proceeding with backup/restore?
It is recommended to put your site into Maintenance mode and create a Site backup.
As long as the data directories for production and dev are unique ... separate (same for DB's and code), yes, you can run the CLI only backup.php script.
Takes web service out of the loop ... and it will report errors as well ... which will give you some clues.
Just make sure you give a destination directory outside of sea of files in moodledata/filedir/
'SoS', Ken
Thanks for explaining ... now we can discuss/cuss options ...
RAM ... 45G/96G ... I take it means 45G for web server and 96G for DB server?
What you didn't supply is free space specs on web server. Am assuming the NFS you mentioned has no space issues and plenty of free space remains.
Trying to setup your own task in a cron job separate from moodle's cron job to clean out moodledata/temp/backup/ is almost impossible to get the timing right. Your clean up cron will probably romp on a auto backup in progress.
So let's talk about space ... the largest directory in moodledata is filedir. That's where moodle stores files for the most part - and where moodle will store .mbz files if not told otherwise.
Other directories in moodledata ... like caches ... are best served quickly if they remain on the same server as code .... faster responses.
One could do this ... all of moodledata remains where it is and only filedir is on NFS via a mount point on web server. moodledata/filedir/ looks to be local drive but it's actually a mount point that goes to /nfs/moodledata/filedir/ Yes, that will be a bottleneck (slower) but ....
Factors ... preferences for backups and preferences for auto backups ... the first I would imagine are set to backup everything in a course. Any thing one could really do without? Ditto autobackups .... surely you are not keeping all - keep 2?
Think I might have used a term (dups) that was mis-communicated.
In moodledata/temp/backup/xxxxxxxxxxxxx/ when backup completes successfully, there is an .mbz file named 'backup.mbz' (that's the dup). But if you left backup file naming to defaults, the backup.mbz file when it is copied to destination, will also have it's filename changed ... something like: backup-moodle2-course-139-testuno-20200925-1800.mbz. So there will not be any dups ... the numbers at end of filename are a date/timestamp.
'SoS', Ken
I guess am not explaining well enough ...
"Auto Backup Starts > backup one course > store in temp/backup > compile > copy to NFS"
Terminology aside, yes, that is what happens with one addition ... after backup has been copied to destination, the actual routine is to erase the temp directory used to build the backup.
Catch 22 ... a build of a backup.mbz file fails or *copy* of backup.mbz fails - for whatever reason ... since autobackup code doesn't get notice that course backup has failed - database tables related don't get a completed ... nothing removed. However there is a clean up task (part of moodle cron) that is supposed to clean up.
Depending upon version of Moodle, an adhoc_task.php script attempts to pick up where things left off - no completion notice.
"Is there a way to delete the temp/backup's copy in real time right after it has been copied to NFS?"
See above!
If you do any other/outside of Moodle cron/task jobs to remove whatever is in moodledata/backup/temp/ I don't know how you can do that without disrupting an autobackup during a build. If anything, you should be running the clean up task ... not a home grown cron to clean up.
How long each of those 1000's of courses takes depends upon each course. I've never seen any documentation that says if a course is X Gig in size it will take Y minutes to backup and copy to destination.
With 1000's of courses I doubt autobackups complete in one day.
Since you have done designated directories, can you determine the largest backup file in that designated directory? Actually verify that what you say "256G in backups is only 1 day's worth" is true? Both in meta data from DB and physically in designated directory?
Now while all this is going on ... are your teachers deleting modules in their courses? If so, those deletions create recyclebin backups which are on a schedule of their own when it comes to keeping, moving to trash, and then eventually removal. Plus, those course backups made by auto can't be accurate any more and might actually ballon a backup if there are recyclebin backups associated with the course.
Also remember, if auto takes all day or 3 days ... what of the 'normal' daily traffic to site? Ever max out memory of code server?
'SoS', Ken
"Terminology aside, yes, that is what happens with one addition ... after backup has been copied to destination, the actual routine is to erase the temp directory used to build the backup."
Okay, good to know that the routine erases the copy in temp/backup right away.
Thanks so much Ken <3.
I'll deal with the failed backups next week after I've setup the NFS for production. (I'll probably scour previous posts on how to deal with them)
P.S. If I could ask a follow-up question.
Right now the setting for Automated backup storage is set to "Course backup area and specified directory".
So does this mean that there are two backups made for the same course?
If I set it to "Specified Directory for Automated backups", will it delete the duplicate in filedir?
"Right now the setting for Automated backup storage is set to "Course backup area and specified directory" ... "does this mean that there are two backups made for the same course?"
Don't mean to be a smarta-- but what do you think?
2nd question ... no ... changing destination options will not automagically remove auto backups that are in moodledata/filedir/
For that ... query of DB to find contenthash value of .mbz files in filedir.
Contenthash value is location in moodledata/filedir/
Manually remove file and then row in mdl_files that refer to those.
For how to do that ... time for you to 'grow' and dig and learn ... all by yourself ... other wise, where do I send the bill for consultation? ;)
'SoS', Ken
For how to do that ... time for you to 'grow' and dig and learn ... all by yourself ... other wise, where do I send the bill for consultation? ;)
Hehe.. Thank you so much for your patience with me Ken. I greatly appreciate it.
Stay safe and God Bless. <3
Hello,
Just noting that in Moodle 3.10 there is also a CLI script for restoring large backup files - MDL-69307.
So are you ready to try the backup.php file in code/admin/cli/ yet?
'SoS', Ken
Without knowing what's in the course ... knowing only that if you do backups selecting say section 1 -> 3 and no more, that works ... how about a different approach that might discover where the issue might lie?
Do a GUI backup and this time de-select all quizzes in the course ... when it comes to naming the backup insert -noquiz- in it's name. If that works, make another backup and this time de-select all other resources in the course *except* quizzes and in the name of the backup insert '-quizonly-'.
Do both fail? Does one succeed? Which?
If one had debugging turned on while attempting above might get some more info.
'SoS', Ken
Good Day!
I had encountered an irrevocable problem today.
I tried to restore a backup and wanted to merge it into the current course. But instead of merging, I misclicked to delete before merging. It turns out, that action also deletes the previous automated backups.
I think this behavior does not make sense. I couldn't find a way to undo my mistake. I wanted to restore a previous backup to perform the restore and merge properly but the previous backup has been removed as well. I checked the server files and recycle bin and they're all gone.
Regards,
Francis Joseph Serina
If one acts soon, one might be able to recover a backup of the course.
While I'm showing command line, you will have to translate that to whatever tools you have ... phpmyadmin for queries ... cpanel or other file browser for files:
To see .mbz files in meta data of database, query the mdl_files table:
select filename,component,contenthash from mdl_files where filename like '%.mbz';
that should list/show something like:
+--------------------------------------------------------+--------------+------------------------------------------+
| filename | component | contenthash |
+--------------------------------------------------------+--------------+------------------------------------------+
| backup-moodle2-course-3-mdlcli-20200623-0912-nu.mbz | mod_resource | 4179aac1c0863c9b27130569c886664ae93c33d6 |
| backup-moodle2-course-3-mdlcli-20200623-0912-nu.mbz | mod_resource | 4179aac1c0863c9b27130569c886664ae93c33d6 |
| backup-moodle2-course-3-mdlcli-20200920-1606-nu.mbz | user | 691dc986b6142c776f4ee296dce7135e3c5aaee1 |
| backup-moodle2-course-136-mdlsampler-20190910-0027.mbz | mod_resource | 7076296142de12818f4b55e692dbb8a32452b319 |
| backup-moodle2-course-3-mdlcli-20201118-0854-nu.mbz | user | bc4e20b32f6ed86d0eec14fdaeb0b60a0432d778 |
The contenthash column info is the location of the file in moodledata/filedir/
Using the last one listed above as an example, your course backup was called backup-moodle2-course-3-mdlcli-20201118-0854-nu.mbz
the contenthash for that file is bc4e20b32f6ed86d0eec14fdaeb0b60a0432d778
From command line or from file. browser like in cPanel:
cd /path/to/moodledata/filedir/ (see config.php file of moodle for location of moodledata directory)
cd bc/4e
ls -l bc4e20b32f6ed86d0eec14fdaeb0b60a0432d778 (list file and shows size of file)
IF command: file -b bc4e20b32f6ed86d0eec14fdaeb0b60a0432d778
shows
gzip compressed data, from Unix
It is a moodle backup. cPanel might have some button for mimetype.
**Copy** it out to some location outside of moodledata/filedir and name it the same as the filename as seen in the DB query:
cp bc4e20b32f6ed86d0eec14fdaeb0b60a0432d778 /home/myaccount/backup-moodle2-course-3-mdlcli-20201118-0854-nu.mbz
where /home/myaccount/ is a directory you can see in file browser and from which you can download.
Download backup.
Then restore via Moodle admin interface.
Good luck!
'SoS', Ken
