Actually, where is that setting to allow compression?

Moodle docs says

"Backing up courses larger than 4GB

Site administrators can enable a compression format for .mbz files (internally stored as a .tar.gz file) from Site administration>Development>Experimental>Experimental settings which removes the 4GB backup size restriction and may improve performance. If this box is checked then future courses will be backed up in this format."

But, I only have 3 settings under experimental on my 2.9 sites, 

- Enable Safe Exam Browser integration

- Drag and drop upload of text/links

- Enable CSS optimiser

What have I missed?

Most of us are wondering why the Experimental is experimental because it is actually a far better format and works much better!!

Compression is only done during the backup.   2.9 has that as default now.

See:

https://docs.moodle.org/29/en/File_system_repository

The only way to avoid download from one Moodle server and then upload to another is to have command line access on the server where the backup is being made.

Run the backup.php script in moodlecode/admin/cli/ and point that script to a directory to which you have shell access ... something like /home/ken/

Then, ssh into the server where the backup was made and go to the directory where the backup was saved.   scp the .mbz file to the other server's file system repository.

If on the same server ... ie, backing up a course to restore to a new course on same server, the process is similar.   Run backup php script with the destination to the file system repository setup for the server.

One can then simply setup the file system repo in any course and then use restore there to restore the course.

Uploading/Downloading files can present some problems ... timings on scripts, upload size in not only PHP but Apache + any restrictions the hosting provider might impose depending upon the package purchased.

'spirit of sharing', Ken

Ken thank you, this is very useful indeed.

The File System repository is great if you can do the copy yourself on the server, but is also the answer to the question of how to make use of FTP to transfer the files, and so avoid Moodle for the upload step where I'm getting the timeout.   

https://docs.moodle.org/29/en/File_system_repository_FAQ    may be useful to others alongside the link you have provided.

Thanks also for confirming the compression is there by default in 2.9.  Moodle docs probably needs updating to reflect that.

Ken has confirmed that in 2.9 it isn't experimental any more and is the default !

I hadn't missed anything.  The documentation is slightly out of date and Ken has confirmed that in 2.9 there is no such compression setting because it is now the default.

If you provide information concerning your setup ... remotely hosted, shared system (also provide hosting provider name), etc. then answers could be more specific or at least find, for you, the FAQ for customers on those hosted systems.  Otherwise, the responses are educated guesses which still might be in-correct.

In Site Admin Menu -> Server -> PHP Info you will see the settings related to time outs.

Doesn't sound like you are on a dedicated server, but rather, a shared system?   Thus, you'll have to consult with provider docs/helpdesk on how to increase time limits and other settings (like size of file to upload, etc.).

FTP ... depends upon IF server is set up to accept FTP connections.   If so, you'll need a login/password (those aren't related to Moodle) and then you'll have to discover the path to your moodledata directory ... where the repository directory resides.  Again, on a shared system, you'll have to consult with provider for specifics.

Now I might be getting confused here ... you are trying to restore a backup made on the SAME server, is that right?   Should NOT have to download to restore a course.  In any course, click restore and see if you can find the backup (private files?).  If you can see the backup, you should be able to restore.

Two Moodle instances on same server and you are trying to restore a course from one instance to another?   Is that right?  IF on same server and saving backups to a designated area (not inside the moodle file system), then whatever you have to browse files and will allow you to copy files.

'spirit of sharing', Ken

I have looked into this a bit further. It turns out that none of the courses will back up, not just the big ones. They always stall in the last few % of the process, the process just hangs there. It does create a directory in moodledata/temp/backup/ but there is no .mbz file there but a lot of .xml files! (see pic).

Is there a bug in the backup routine? Or perhaps I need to change some permissions on a directory on the server? I don't need to download these files from the server, I want to use them to clone the course on the same server.

Are there any other strategies or suggestions that I can try for solving this issue?

Thanks very much!

Looks like Linux (Ubuntu) and you have command line access .... Good!

How much disk space do you have free?

Let's try taking Apache out of the loop with backup.

You can safely remove all files folders in moodledata/temp/backup/

1. Get the course ID of the course you cannot backup.

2. cd /pathtomoodlecode/admin/cli/

3. php ./backup.php --courseid=# --destination=/somedirectory/wherethereisplentyofspace/

In another terminal session, go to your data directory/temp/backup/

When you begin the php script to backup the course id, look in that other terminal session window

There will be a contenthashed named directory created ...

cd thatcontenthashednameddirectory

type: watch "ls -l;df"

and you can see what Moodle is working on ...

Can tell you right now files are numerous from what you show above that will take the longest (largest subdirectory/largest .xml file).   So when the backup routine hits the part for files, it will slow down and take time ... even the command line might appear to stop.

BTW, the 'course' directory isn't really the course!

Wonder what another terminal session running top would show?

'spirit of sharing', Ken

I have a course I'm trying to restore that has a large question bank. Restoring the course takes over 4 hours. if I exclude the question bank the course can be backed up and restored within about 10 minutes. I've seen a few tracker tickets but none reference 2.8. Is there a tracker ticket regarding this or does anyone have a work around or solution?

Thanks.

I wish I had a solution but I do not.  We are struggling with large question bank restores as well -- though not as terrible as 4 hours!  We feel your pain though.

We recently found moosh http://moosh-online.com/commands/#course-restore and are interested in testing it to see if it does any better.

-- Scott

Hi Ken and everyone

First post on Moodle forums.... thanks to everyone for the advice across so many topics!

Quick question on backups.

We're using Moodle 2.9 on hosted service. Course backups getting out of control and now eating up Gb of filespace every day and growing in consumption each day. I've identified the rogue course (few hunded Mb of files, and am working on a longer term solution) using the advice on here and trying various ideas out.

Meantime, have suspended course backups within moodle until I can get a long term solution (still have daily server backups going though)

My question: is it OK to delete moodledata/temp/backup by just going in there and delete via the file manager, or do we have to delete using a Moodle script (read somewhere that everything is now tied to the DB....). Our temp folder is now enormous with unfinished / failed backups and I'd like to just blitz it just to get control of my disk usage.

Sadly, this is on our production installation, and the problem only started when a single course uploaded about 500Mb of course materials post go-live (but that's an aside), so it is urgent as our disk usage is rising rapidly.

Thank you all in advance

Ed

I'm mulling through this...thanks again. A bit more info/clarity on what I recovered during the mess up going to Ubuntu 15.  

In var/lib/mysql...

First, pardon the length  of this ... answer is not so simple. :\

Second ... am by no means the foremost expert on the backup process, have just had to deal with it (like you) and have found what works for me ... your mileage might vary.

Before removing all files in moodledata/temp/backup/ it's worth the time to investigate what one finds there.
A successful backup leaves behind a 0 byte log file.

All commands below are from moodledata/temp/backup/ unless otherwise noted.

ls -l *.log
renders a list such as this one example:
-rw-rw-rw- 1 apache apache    0 Aug 27 15:36 a3afdfe6e44507913ed73609fbb1509b.log

Nothing in them so one can erase them:
From moodledata/temp/backup/
rm -fR *.log

That leaves one with only directories of failed backups.   They are named with a contenthash.

ls -l ./*/*.mbz

To see if there is a moodle backup file (.mbz) contained in any of them.

If there is, that file could be a valid backup that couldn't be *copied* from the build area
to the destination as determined by the backup settings.  Destination could be 'backup area' ... which is the moodle file system itself in moodledata/filedir/ ... or 'designated' ... which is a directory the op has to
create manually and set proper ownership/permissions upon so that Apache can write to it ... or 'both'.
Uhhhh ... a system that is having issues with auto backups probably shouldn't be using 'both'.
From what I've been able to determine, if there is a .mbz file in a temp directory, the failure occurred
at the very end of the backup process ... which is to 'copy' the .mbz to the location.   Copy on systems
have a limitation on file size and the only way to increase it is to compile support for larger files
in the kernel.   Uhhhh .... something most folks won't do.

However, one can use the 'move' command.  It has no such limitation.  So if you have found .mbz files
in any of the temp/contenthash directories, they might be good backups ... just couldn't be copied.
** Move them to a designated directory for testing later (testing is simple, see if you can un-tar-gz them).

Example:
mv *.mbz /pathtosomelargepartition/

Example of one failed backup below.   The directory 58.... below contains a failed backup.

[root@moodle backup]# ls -l
drwxrwxrwx 6 apache apache 4096 Aug 24 14:06 5860eac2e0ff3537538cf5fcaa0a84b8

cd 5860eac2e0ff3537538cf5fcaa0a84b8

And then list files contained therein:

[root@moodle 5860eac2e0ff3537538cf5fcaa0a84b8]# ls -l
total 96
drwxrwxrwx 14 apache apache  4096 Aug 24 14:06 activities
-rw-rw-rw-  1 apache apache    79 Aug 24 14:06 completion.xml
drwxrwxrwx  3 apache apache  4096 Aug 24 14:06 course
drwxrwxrwx 28 apache apache  4096 Aug 24 14:06 files
-rw-rw-rw-  1 apache apache 31251 Aug 24 14:06 files.xml
-rw-rw-rw-  1 apache apache   215 Aug 24 14:06 gradebook.xml
-rw-rw-rw-  1 apache apache    86 Aug 24 14:06 groups.xml
-rw-rw-rw-  1 apache apache     0 Aug 24 14:06 moodle_backup.log
-rw-rw-rw-  1 apache apache 17684 Aug 24 14:06 moodle_backup.xml
-rw-rw-rw-  1 apache apache    83 Aug 24 14:06 outcomes.xml
-rw-rw-rw-  1 apache apache    83 Aug 24 14:06 questions.xml
-rw-rw-rw-  1 apache apache   294 Aug 24 14:06 roles.xml
-rw-rw-rw-  1 apache apache    79 Aug 24 14:06 scales.xml
drwxrwxrwx 15 apache apache  4096 Aug 24 14:06 sections

In the example above, NO .mbz file so even though it began to build, failure for some reason.   Note the moodle_backup.log file is 0 bytes.  No help there.  But, you can see files.xml is rather large - those .xml
files, BTW, are like 'controllers' for their respective directories.

In the above example, it's in-complete and probably could not be used in a restore, but ...
for the sake of attempting recovery o
f something .... we'll make a backup file manually.

To get the course name, from the moodle_backup.xml file:

fgrep '<original_course_fullname' moodle_backup.xml

In this example, the command above renders:
<original_course_fullname>Freshman Leadership</original_course_fullname>

Now we know what to name the backup we are creating manually.

From the 5860eac2e0ff3537538cf5fcaa0a84b8 directory:
tar zcvf /somelargepartition/freshmanleadershipmanualbackup.mbz ./*

Now it might not be a good backup ... test later.

Since I'm using a system that did have such a course as a failed backup, will continue using
the above example.

We need to find out if that course is still there.

In the moodle_backup.xml there is a line:
<original_course_id>244</original_course_id>

That's the id number - 244 - I'll use that below.

Now let's see if we can back it up via command line.   Must have that ID number.

Before we leave the temp area, let's remove any all files/folders in there.
From moodledata/temp/backup/
rm -fR *

Ok to test backing up that one course via command line.

cd /pathtomoodlecode/admin/cli/

There is a backup.php script.  It will use parameters from the setup of course backups
in Site Admin -> Courses -> Backups -> General backup defaults.
IF one is including users, backups will take longer.  IF excluding users, the backup
file name will contain 'nu' (no users) in the file name.

On a failed course backup, I'll set that to no users one time and run the command line
backup to see if that will complete.   Then I'll change the setting back to include
users and try again.  Obviously, including users should result in user data being included
AND the backup will take longer and BE larger.

Issued like:

php backup.php --courseid=244 --destination=/home/clibackups/

where 244 if the course ID and in this case I have a directory called clibackups on the
largest partition of this system called /home.

I'd work in two terminal sessions at this point.
One terminal session (#1) where the command is issued and runs.
The other (#2) in /pathtomoodledata/temp/backup/

Once you issue the command in #1 switch to #2 and see what contenthash directory
has been created to build the backup.  In #2 issue:
watch "ls -lR;df"
and that watch command will show what Moodle is doing as it builds the backup.

The command line backup script, won't display much:

== Performing backup... ==
Writing /home/clibackups/backup-moodle2-course-244-fl-20150829-0809-nu.mbz
Backup completed.

IF successful.

[root@moodle cli]# ls -l /home/clibackups/backup-moodle2-course-244-fl*
-rw-rw-rw- 1 root root 4275819 Aug 29 08:09 /home/clibackups/backup-moodle2-course-244-fl-20150829-0809-nu.mbz
-rw-rw-rw- 1 root root 4282650 Aug 29 08:19 /home/clibackups/backup-moodle2-course-244-fl-20150829-0819.mbz

You can see the nu backup is smaller.

But your #2 terminal session window will show when/where it fails.

What to do at this point, I couldn't tell you.   But, it might point to something
or provide a clue.

I'd set auto backups to run manually.

As far as discovering what courses are the largest, thus the ones that would/might give grief, there
is a a report for courses sizes one could install.  Once installed and run, you can ID which courses
to try out the command line backup upon.

To get control over it all, one might have to create the command line scripts that
go through a list of smaller course ID's and runs the command line version of backup.
And another script that runs through a list of larger courses on another day - the least used
time frame.   Those scripts can be setup with cron.

One might consider putting the site in maintenance mode when running the larger courses backup script as those backups ARE resource intensive.

I can provide examples later.

Ken

Am having to work on this very thing this AM and just discovered a little thing to help see what's going on in the build area of the backup.

In moodledata/temp/backup/

while watching Moodle do it's thing issue the following watch command:

watch "ls -lR -I '*.log';df"

After the ls -lR that's a dash capital "I" ... not a lower case L. I stands for "ignore'.

That will ignore the .log files and show only the contenthashed named directory Moodle is using to build the backup.   At the very top of that contenthashed directory, the .mbz file being built will show 'progress' ... ie, you'll see it increase in size.

You could drop the ;df as that would show disk free output and it's assumes you are working with a LARGE disk.  Screen clip follows:

'spirit of sharing', Ken

Thanks for that - lots to be working through from a diagnosis/resolution pov.

But in essence, outside of solving the root cause, it's OK to just delete the contents of the temp/backup folder as its contents are not referenced anywhere in the database. That way I can get the disk usage back under control.

In the checksum folders  have a backup.mbz file of 800+Mb, so it would point to the "copy" issue you speak of, and am on a hosted service, so hands somewhat tied.

Yes, removing files in /moodledata/backup/temp/ will free the space up and there shouldn't be a reference to that file in the mdl_files table ... didn't finish.   Of course, am not the programmer of the routine so don't know exactly when the routine records info into DB.

An sql query of mdl_files for file names that end with .mbz should be the valid backups.

Do you have the option turned on for use course name in the backup file name turned on?  Helps.

Am not sure ANYONE could give you a definitive answer without really knowing a lot about the remotely hosted system and the courses and course usage themselves.  

Even if you turned on debugging .... automated backups won't have any screen to show errors and if there was a way to grab those errors, there becomes the problem of translating them into humanly understandable language!

Apache error logs *might* have some clues ... but I've seen remotely hosted systems setup where if something gets over limits, it kills whatever script AND doesn't report any error to the 'customer' ... that's something nasty but suppose it is assumed the OP knows what limits customer has purchased.

More and more am thinking sites of certain levels of usage need to be on dedicated server and one where the op has command line access.  OR host with Moodle Partner ... OR get some 'semi-expert' help.

Afraid this sort of thing requires sluething.

I run/help admin RHEL/CentOS 5,6 servers - they have that can't *copy* larger than 4+Gig issue.  And I have had to use command line and use 'move' to get those backups.

Situations are such that I cannot tell the digital photography teacher NOT to make camera assignments to 125 students requiring students to upload 6 photo types and for each photo type of 6 photos.   That's just one assignment!   See what I mean?

How about the teacher who decides to attempt a flipped classroom using nothing but Moodle?

Sooooo .... ????  We do what we must to support.

'spirit of sharing', Ken

In /var/lib/mysql/ is there a mysql_upgrade_info file?

cat mysql_upgrade_info

what does that render?

What does: mysql -V display?

Something like this?
mysql  Ver 14.14 Distrib 5.5.44, for debian-linux-gnu (x86_64) using readline 6.2

Just looked at a Ubuntu box and here are the files in the /var/lib/mysql/ directory:

root@Moodle2:/var/lib/mysql# ls -l
total 3964780
-rw-r--r-- 1 mysql mysql          0 Aug  4 08:09 debian-5.5.flag
-rw-rw---- 1 mysql mysql    5242880 Aug 30 00:25 ib_logfile0
-rw-rw---- 1 mysql mysql    5242880 Aug 27 15:54 ib_logfile1
-rw-rw---- 1 mysql mysql 4045406208 Aug 30 00:25 ibdata1
drwx------ 2 mysql mysql      20480 Aug  4 08:13 moodle
drwx------ 2 mysql mysql       4096 Aug  4 08:09 mysql
-rw-rw---- 1 mysql mysql          6 Aug  4 08:09 mysql_upgrade_info
drwx------ 2 mysql mysql       4096 Aug  4 08:09 performance_schema

All the files in the moodle directory ARE .frm files cept the db.opt so that's 'normal'.

From the mysql> prompt, what do you get for this query:

mysql> SELECT `ENGINE` FROM `information_schema`.`TABLES`WHERE `TABLE_SCHEMA`='moodle';

If all the tables in the DB are InnoDB you'll get a listing of InnoDB for however many tables in the moodle db there are.  Approx. 316 rows in set.

Can you run a mysqldump?

Ken

FULL RECOVERY!!!--I can't believe it.

Kept getting database errors re: timing out and other issues due to knowing nothing about DBs...so did a lot of experimenting. I used all the help in this string plus heaps of reading up on mySQL recovery, permissions for mySQL folders and files, and mySQL DB users/permissions. I'm not 100% sure what made the difference in the end.

Here is roughly how the fix unfolded, not too sure if this was the order or if I missed anything.

• Fresh install of Unbuntu 15.04 on second VPS because that is what I was upgrading to when I crashed. I am clueless if this mattered but I thought the less variables the better.
• Installed mySQL
• I think I added innodb_force_recovery = 6 under [mysqld] to mysqld.cnf. Struggled knowing which .cnf file to use because it should have been my.cnf? But got errors with that one. 
• I probably forgot to "service mysql restart" after changing .cnf file...so remember to do this.
• I think I copied over the recovered moodle folder with .frm files and the recovered ibdata1 into /var/lib/mysql. I'm pretty sure I didn't copy over the lof files..maybe even deleted them?
• Don't think I did this but may have also put innodb_log_file_size=XXM, with the "XX" as the size in mb of the original log files I think?
• To make sure the permissions were OK I temporarily made all of the files and folders 0777 with mysql as group/owner
• Kept having trouble with these:
  
• • sudo mysql_upgrade -u root -p --force
  • mysqldump -u root -p moodle > moodlefixed.sql
• So did this and the dump worked.

Take home point to avoid spending 50 hrs on this (but learing a lot along the way): you need to back up moodle, moodledata, and you need to backup the database /var/lib/mysql properly by mysqldump:

Congrats!  Know that it's been an 'unpleasant' learning experience but you've internalized much in a short time and learned the most important lesson of backing up Moodle the easiest way.

From mysql docs:

If you change a global system variable, the value is remembered and used for new connections until the server restarts. (To make a global system variable setting permanent, you should set it in an option file.)

Now might I offer little bash shell scripts to facilitate backups in parts?   I have 3 that backs up code, the data directory, and the DB.   Must manually create the backup directory.  I usually use /home/backup/

As root user:

cd /usr/local/bin/

nano [nameofscript]

echo 'backupcode';
tar -cvf /home/backup/moodlecode-$(date +%Y%m%d).tar /var/www/html/moodle;
ls -l /home/backup/moodlecode*

echo 'backupdata';
tar -cvf /home/backup/moodledata-$(date +%Y%m%d).tar /var/www/moodledata;
ls -l /home/backup/moodledata*

echo 'backupdb';
mysqldump -u root -p[password] moodle > moodledb-$(date +%Y%m%d).sql;
ls -l /home/backup/moodledb*;

To do it all now, from any location, as root user, issue:

backupcode; backupdata; backupdb

If updating, upgrading or installing plugins just run:

backupcode;backupdb

'spirit of sharing', Ken