I'm backing up and restoring courses from moodle 3.0 to moodle. 3.4 both on ubuntu servers.  

I've used Moosh on both and when I try to restore the course is always blank.  I tried to restore to the same 3.0 Moodle server just as a test and it's blank there too. I'm using:

moosh -n course-backup -f /tmp/mybackup.mbz 77

checked the defaults for backup and selected variously but always include activities and questions and still it imports blank courses. 

Checked importing defaults on moodle 3.4 and all are selected so I should not have an issue.  

I read Ken's suggestion to use nohup here and wondered if this technique works with 3.0 and above? or if it would make a difference.

ex, nohup php ./backup.php --courseid=INTEGER --destination=STRING

I've read it can be many other issues at your link here but that list is so long I don't know where to begin. 

I know you said there are not many flags and I saw the -e flag in backingup but I think that means to an EXISTING course not everything.  It takes the defaults as I've read and I have played with defaults with no luck. Thanks for advice.

The nohup affects any shell command issued ... no hang up ... complete the script ... period.  I use it only when the ssh server timeouts kick in before the backup/restore commands have a chance to complete.

The other thread you referenced dealt with an addon in a course and the poster did find a hack that fixed his issue .... relating to the addon/plugin.

The restore begins with the backup ... so do you have addons/plugins active in the course you are trying to backup?   What if you created a backup, via the Moodle Admin UI, set backup preferences to a directory outside of Moodle filedir and excluded anything/everything that was non-core (ie, no addons/plugins)?

On the 3.0 server ... have you run the hidden/still experimental health check?

https://site/admin/tool/health/

That does find issues with Quizzes and does offer SQL suggestions on how to fix.   If you find any, run the SQL fix.   Any of those Quizzes in the course you are trying to backup via moosh and restore to higher version server?

I would think that using command line moosh, if there were an error it would display or one could setup php to log errors.  In one shell window, run the moosh command.  In another shell window, run tail -f /var/log/php-error.log ... that shows the error log in realtime.

Afraid there is just not enough info (you don't know what you don't know to share) to know and share exactly what the problem might be.

Check PM in Moodle .... I might have an offer you probably shouldn't refuse but do realize the backup contains users that I would be privy to.

'spirit of sharing', Ken

Thanks Ken,

for those following. If I do it manually it works perfectly. two things I noticed, I do have a 3rd party app called Blackboard Ultra that I unchecked before importing. I wonder if that file would make the restore fail on the new moodle?  Of course it should not affect it on the old moodle since that app is installed but you never know.  

The second thing I noticed was a warning, "Role in backup file cannot be mapped to any of the roles you are allowed to assign" 

It was for student which I manually selected and I did notice this error through Moosh too but am not sure which activity has a role assigned to it or if that would prevent the entire restore from working. More info to follow . . .

That plugin ... Blackboard Ultra ... would need to be installed in the new server AND configured to restore a course that used that plugin.   Could cause a hickup in restore.

As far as role mappings ... in the setup of Blackboard Ultra, is there a sudo user ... account ... role setup with whatever Blackboard server with which it interacts?

Page:

https://blackboard.secure.force.com/publickbarticleview?id=kA7390000004F6j

Dunno if I found the right plugin or not ... does say:

Configuring the plugin
Collaborate Ultra plugin takes SAS admin credentials.

You say that was a student account .... can you un-enroll that student account from the troubled course?

'spirit of sharing', Ken

for those following this, I found a Workaround:

 if I import the problem course into a new course but just activities, questions , and poorly edited topics/dates by user (where too many pics are in date) it works to moosh backup on that new course.  That's a bit of a pain to create a new course every time a backup fails but at least it works.

note: I tried moosh backup with and without the plugin Ultra and both worked so that was not the holdup nor the topics/dates but something else.

Jamie and I have been 'collaborating' (not the mod) ... he uploaded an .mbz to a server outside of Moodle using Webmin.

That allowed me to inspect what was actually in the backup by un-compressing it and inspecting what was there.

In the un-compressed backup no user backup uploaded, in activities directory

assign_1302  assign_8608           label_1295    lesson_14329  quiz_1298  quiz_1311  quiz_8607  resource_1293   resource_8603
assign_8589  assign_9184           label_9194    lesson_14330  quiz_1300  quiz_8587  quiz_8609  resource_1303   resource_9181
assign_8591  bigbluebuttonbn_8586  label_9216    lesson_14331  quiz_1304  quiz_8588  quiz_8610  resource_14328  url_1294
assign_8593  collaborate_2735      label_9217    page_8613     quiz_1305  quiz_8595  quiz_8614  resource_8581   url_9190
assign_8596  collaborate_9493      label_9221    quiz_1296     quiz_1306  quiz_8599  quiz_9182  resource_8583
assign_8601  forum_9220            lesson_14327  quiz_1297     quiz_1309  quiz_8604  quiz_9183  resource_8598

All those but 'bigbluebuttonbn_8586', 'collaborate_2735 and collaborate_9493 are stock moodle.  The numbers are the ID numbers of the assignment/label/quiz, etc..

and in the moodle_backup.xml which is a roadmap to restore:

[root@sos sos]# fgrep 'collaborate' moodle_backup.xml
          <modulename>collaborate</modulename>
          <directory>activities/collaborate_9493</directory>
          <modulename>collaborate</modulename>
          <directory>activities/collaborate_2735</directory>
        <activity>collaborate_9493</activity>
        <name>collaborate_9493_included</name>
        <activity>collaborate_9493</activity>
        <name>collaborate_9493_userinfo</name>
        <activity>collaborate_2735</activity>
        <name>collaborate_2735_included</name>
        <activity>collaborate_2735</activity>
        <name>collaborate_2735_userinfo</name>

Can see a 'userinfo' thang ... which I have no idea what it does ... but could have been the source for the user roles conflict.

Course was also fairly 'heavy processing' in quiz:

-rw-r--r--. 1 root root 1234304 Jun 28 18:30 questions.xml

Sometimes courses like this might look small when viewing the size of the .mbz file (only 7723823) but they are heavy processing on restore ... 1.2 megs worth of question data (approx) in quizzes.

One could edit xml files, re-package the backup and then TIA.

All the above NOT the recommended way to fix but when all else has failed it is possible to recoup some of the course.

'spirit of sharing', Ken

Hi,

I'm having an issue where my bulk course backups is eating up gigs of space even though I'm sending the backups to a completely different drive located on a separate NAS device.

I've checked the /moodledata/temp/backup directory and there is a bunch of log files in there but they are not taking up much space. 

Can someone tell me if it's safe to delete the contents of the moodledata/trashdir?.... Besides the moodledata/filedir, the trashdir is the next largest directory taking up 55.2 GB of space. Please see attached screen shot for a list of directories and the amount of space they are consuming.

If it's okay to delete the contents of the trashdir from the command line within Linux will it have any impact on the end users or the system in general. If I delete the contents will it start trowing errors on the GUI side.

Any help is appreciated.

Thank you,

-Sean

Looks like having only 4 Gig space would make it tight. :\  I have manually removed all contents of the trashdir with no ill affects site wide, but, then again, dunno if a teacher attempted to restore a file deleted or not ... none have reported an issue.

But I do have a question ... and this might free up some more space as well ... what are the 'numbered' directories you show in the screen shot?   Those look like the old =<1.9 course ID directories.  The /1 was the front page of =<1.9's.  The /1 can removed for sure as you can't use it to replace front page anyway.   But do look in there to see what's there before removing.

With things that tight space wise ... every little bit helps! ;)

'spirit of sharing', Ken

Problem finally fixed in the latest Moodle releases.

Excellent, Thank YOU!

Natassia

These are our usual scores:

Not something to be proud about... Is there any way how we can improve this score? A 13% succes rate on automated backups is just horrible.

Hi there,

i been facing issue while running script for automated course restoring.

with the reference to the following link

https://moodle.org/mod/forum/discuss.php?d=196596

Error:

D:\wamp64\www\totaralms-9.7\admin\cli>php Auto_Restore_Script.php

Server Time: Wed, 18 Jul 2018 09:41:41 +0530

Starting to extract: D:\wamp64\sitedata/autoexport/backup-course-from-2.7.16.mbz

 to: D:\wamp64\sitedata/temp/backup/00000000026497

!!! error/ !!!

!!

Error code:

$a contents:  !!

!! Stack trace: * line 164 of \backup\util\plan\restore_plan.class.php: restore_

controller_exception thrown

* line 333 of \backup\controller\restore_controller.class.php: call to restore_p

lan-&gt;execute()

* line 68 of \admin\cli\Auto_Restore_Script.php: call to restore_controller-&gt;

execute_plan()

 !!

guys do let me know if anybody have remedy for this...

This past February we changed the automatic backup to go into a specified directory.  So I know where those backups are!  However, when I go to Backup under the Course Administration, I have a backup under "Course backup area" as well as some in the "User Backup Area".   Where are these files kept so we can delete them?

Also, if you do automatic backups, would it be good to not let your teachers do additional backups?

What version of Moodle?

"Course backup area" as well as some in the "User Backup Area" are kept in the 'sea' of files ... moodledata/filedir/  DB query of mdl_files for contenthash then deciphering contenthash tells you the 'path' in filedir/XX/YY/XXYYxsdflwjfqdwjflfdkjdfl  ...  filename contains .mbz.

To get rid of them ... assuming your autobackups are working ... cannot you delete where you see them?

There's also another area ... user private files ... to see those, login as that teacher ... then you can see/delete them.

'spirit of sharing', Ken

Moodle 3.2 was what is was until most recent update to 3.5.

I can delete them as a user, but as an admin how do I delete all of them? Sounds like I can just search for all .mbz  on moodle data and delete those since the current backups are in a place other than moodle data. 

Thanks! 

No tool in core code nor plugin that I've found that will 'delete them all'.

Use query to find contenthash ... remove manually or via script then also remove the mdl_files row/record that referenced that .mbz file.   (not fun, but doable).

Might check out Moosh

https://moosh-online.com/commands/

file-datacheck

Go through all files in Moodle data and check them for corruption. The check is to compare file's SHA to their file names.

moosh file-datacheck

Check that all files recorder in the DB do exist in Moodle data directory.

moosh file-dbcheck

Delete Moodle files from DB and possibly move them to trash. File IDs can be provided as arguments or on the standard input (with moosh file-delete -s). --flush option will remove the trashcan directory.

Example 1: Remove files with IDs 10,20 and 30.

moosh file-delete 10 20 30

Example 2: Remove all files with size greater than 100 bytes

moosh file-list -i 'filesize>1000' | moosh file-delete -s

Example 3: Flush trashcan

moosh file-delete --flush

Example 4: Remove all automated backups and reclaim the space

moosh file-list -i 'component="backup" AND filearea="automated"' | moosh file-delete -s
moosh file-delete --flush

Search and list files from mdl_files table. The argument should be a valid SQL WHERE statement. Interesting columns of possible search criterias are: contextid, component, filearea, itemid, filepath, filename, userid, filesize, mimetype, status, timecreated, timemodified.

You can also use some special values:

• course=NNN to list all files that relate to a course

The output will contain some defaults or nearly all possible file information if "-a|--all" flag is provided. The meaning of the flags column is (in order):

• mdl_files.status
• lowercase letter "d" if entry is a dicrectory
• "e" if external file
• "i" if a valid image
• "m" if time created and time modified differ

With "-i" option only IDs are returned. This can be used when pipe-ing into other file-related commands.

Use the -m option to list files that exsist on the {files} DB table but are missing from the file system, and add -r option to remove them from the {file} DB table.

Example 1: Show all legacy files for a course, which context id is 15

moosh file-list "contextid=15 AND component='course' AND filearea='legacy'"

Example 2: Display full information on file with ID 17

moosh file-list -a id=162

Example 3: Show all files from course 6

moosh file-list course=6

Example 4: Super-combo. Get all course files and tar/bzip2 them up.

moosh file-list -i course=2 | moosh file-path -s -r | tar -C $(moosh config-get core dataroot) -T - -cjf files.tar.bz2

Show full or relative path in the filesystem to Moodle file(s). Files can be identified by ID or hash (auto-detected) as arguments or on stdin (-s option).

Example 1: Show path to a file with contenthash da39a3ee5e6b4b0d3255bfef95601890afd80709

moosh file-path da39a3ee5e6b4b0d3255bfef95601890afd80709

Example 2: Show paths to files with ID bewteen 100 and 200

moosh file-list -i 'id>100 AND id<200' | moosh file-path -s

Example 3: Like above but with no duplicates and show path relative to data root (-r)

moosh file-list -r -i 'id>100 AND id<200' | moosh file-path -s | sort | uniq

Just as a side note, I understand that there is a bit of a push on for doing more things in bulk. As courses grow in number, there is going to be an increase in demand for disk space. This may not have been a big issue in the past, but it is not going to go away. OK, disk space is cheap, but we can't hold on to everything. So I think we can reasonably expect to see things like bulk deletions of older backups, unless you want to create a deliberate archive somewhere for them, in the not too distant future. 

My Moodledata\temp\backup folder has thousands of info_from_mbz_.... folders. It seems like every minute one seems to be created.

In backup_general_helper.class.php you'll find this:

          // Extract moodle_backup.xml.

 244         $tmpname = 'info_from_mbz_' . time() . '_' . random_string(4);

 245          $tmpdir = $CFG->tempdir . '/backup/' . $tmpname;

 246          $fp = get_file_packer('application/vnd.moodle.backup');

Shouldn't this be cleaned up at any time? As seen in this threat our automated backup procedure is running but only has a success rate of about 15%.

Ken Task suggested I should use Moosh earlier, but since this is no regular Moodle plugin I can't use it.

Recommend Moosh cause it's probably a very efficient tool ... but alas, not for those who desire to avoid the command line ... on any flavor.

However, check out your Moodle's scheduled task.   Think there is one in there that 'cleans up'.   If executed, it will clean up but from what you say about your automated backup woes, it might attempt clean up of an active backup process in automation.  But, one does what one has to do if running out of space ... and that's the very last thing one wants to happen in a Moodle server especially if the database server resides on the same drive space.

Any site that is used, has many many courses, and/or has very large courses will face issues with automated backups me thinks.

Anyhoo ... check out the scheduled task on your system

'spirit of sharing', Ken

Oh, I have no objection against command-line. Hell, I love the 80's. 

It's just that in many cases the one administrating the server (let's call him John) and the one administrating Moodle (let's call him Pete) is not the same person.

Now John does not want Pete to have command line access to the server. And Pete does not want John to have full admin rights in Moodle. John works for an IT company in another country then Pete, who works for the school using Moodle.

This setup is what I see everywhere at bigger schools (universities).

You could call it a luxury position to be in if you have your hands in everything: database, php, linux, Moodle, education, etc. But it also makes your organisation more vulnerable if that one person administrating everything happens to leave or get sick.

-----

Back to the automated backup procedure... the clean up script probably only runs after the whole automated backup sequence has finished. For us, to run the complete automated backup sequence takes about one month. 

During this month many, many temporary files will accumulate in moodledata\temp\backup

I'm talking about around 9000 folders and files. Not to big though, it does not crash Moodle yet.