If John is in control of the server, then he needs to come up with a script to take care of it!  Does Pete not have Cron access either?

But if these are automated backups, then you can change the setting to only keep one or two copies.  However, if the process would indeed take a month to complete, then it is never going to complete, so turn it off and come up with another backup policy...

Sure I can ask John to make a quotation for a backup script (remember... he's working for a foreign IT-company). But bare in mind John has no Moodle knowledge whatsoever.

The server is being backup-ed, so no worries there. It's just that a server backup takes a long time to restore as well when you have a big Moodle installation.

What I'm dealing with is outside the red area and I do understand that this is not Moodle's focus.

On many occasions I'm just running into problems only sites with 5000+ users seem to have. This is just one of them. I have never seen automated backup working on sites outside the red area.

I love Moodle, don't get me wrong, Moodle seems to be perfect for small(er) schools... it's just that I run in many limitations where things are just a little more complicated that the school-server.

We could share stories about the politics of technical/non-technical roles/access/ etc forever.  Even cross over knowledge ... just enough to understand the affects of what one does on other un-forseen(?) areas of the server.  

What you've described happens in many industries ... Corps, Higher Ed, Lower Ed, and K12.

But let's focus on the technical issues ...

In these 9000 directories are there any .mbz files?

Have you check the tables that control automated backups scheduling to see if they haven't gotten confused ... date/time of next execution column?

Where are automated backups going?   Designated directory or moodledata or both?

'spirit of sharing', Ken

So... I messed up. Didn't backup Moodle and have been trying to restore for the last three days using the following;

/var/www/html/moodle

/var/www/moodledate

And all the files within /var/lib/mysql/moodle

Varied success I have now made progress and got the attached error. /var/www/html/version.php shows $version = 2018051700.00

I have phpmyadmin installed. According to another forum post to get past this error I need to edit mdl_config within the moodle database and edit the version so they are both the same.

Can someone please point me in the right direction in doing this?

Thank you in advance.

P.s as you can imagine this is a very stressful time under a lot of pressure to get this back up and running.

If you took no backup, where did the folders/files related to moodle come from?

Where did the files/folders in /var/www/html/moodle come from?

Where did the files/folders in /var/www/moodledata come from?

Where did the files in /var/lib/mysql/moodle come from?

The files in /var/lib/mysql/moodle  are the raw database files for the Moodle DB.  They should be owned by the user under which MySQL runs (assuming mysql DB as you've not said) which is 'mysql' typically.

Those files should have special permissions in that only the owner has read/write/execute. and the group has 'read' permissions.

Files there look like this example of the last tables for a 3.5 moodle:

-rw-r-----. 1 mysql mysql   9306 May 17 11:57 mdl_workshop_submissions.frm
-rw-r-----. 1 mysql mysql  73728 May 17 11:58 mdl_workshop_submissions.ibd

All those files one cannot edit with anything but a DB interface ... such as phpmyadmin.

In the moodle code you have ... /var/www/html/moodle/ there is a version.php file.  From what you've shared it's a version 3.5 of moodle?  That's a guess ... is it?  Please verify that.

In a version.php file of a 3.5.1 this following is a clip of that file:

$version  = 2018051701.00;              // 20180517      = branching date YYYYMMDD - do not modify!
                                        //         RR    = release increments - 00 in DEV branches.
                                        //           .XX = incremental changes.

$release  = '3.5.1 (Build: 20180709)'; // Human-friendly version name

$branch   = '35';                       // This version's branch.
$maturity = MATURITY_STABLE;             // This version's maturity level.
[root@sos moodle35]#

In PHPMyAdmin, the table is mdl_config the name column set to version, displays this via mysql> shell:

mysql> select id,name,value from mdl_config where value like '2018051701';

+----+---------+------------+
| id | name    | value      |
+----+---------+------------+
| 28 | version | 2018051701 |
+----+---------+------------+
1 row in set (0.00 sec)

What does yours show?

I think that's what the error refers.

You should be able to edit that row ... BTW, your ID may or may not be ID 28.

Fingers crossed for ya, cause this doesn't look good with no proper sql backups of the DB. :|

'spirit of sharing, Ken

I managed to access the failed server locally and recover a copy of the /var/www/html/moodle, /var/www/moodledata and database folder within lib.

/var/www/html/moodle/verison.php states $version = 2018051700.00;

When running query within phpmyadmin I get the following error #1932 - Table 'moodlenew.mdl_config' doesn't exist in engine.

Yet when I look under the database structure I have a mdl_config file

defined('MOODLE_INTERNAL') || die();

$version  = 2018051700.00;              // 20180517      = branching date YYYYMMDD - do not modify!

                                        //         RR    = release increments - 00 in DEV branches.

                                        //           .XX = incremental changes.

$release  = '3.5 (Build: 20180517)'; // Human-friendly version name

$branch   = '35';                       // This version's branch.

$maturity = MATURITY_STABLE;             // This version's maturity level.

What operating system?

Was the database on the old server named 'moodlenew' or did you change the name of the directory from 'moodle' to 'moodlenew' manually?  Did you do that with mysqld service running?

When you acquired the raw db directory/files for moodle on crashed server was the mysqld service running?  Try to re-acquire the db files with service stopped.

If you have changed the name of directory manually,  stop the mysqld service.  Then make the directory name match what you see in config.php file of moodle code.   Also check ownerships/permissions on not only the directory you renamed, but all files recursively.

Do you know how the moodle crashed?  Was the crash originally a DB error?

On the old server, moodle may not work, but what of mysqldump or
phpmyadmin (from which you could get an sql dump)?

Never having been in your situation, the following is google'd research.
https://www.google.com/search?q=Table+doesn%27t+exist+in+engine

Several links ... stackoverflow, dba.stackexchage, etc.

From what I gather of advice in links below, the MySQL server should be stopped
before working with raw database files transfers.  So on the 'crashed' server,
stop the mysqld service,   Then get a tar ball of your moodle DB directory.

The following from a Linux server running
Server version: 5.7.23 MySQL Community Server (GPL)

[root@sos mysql]# ll
total 221608
-rw-rw----. 1 mysql mysql       56 Mar 23  2017 auto.cnf
-rw-r-----. 1 mysql mysql    26454 Jul 28 10:52 ib_buffer_pool
-rw-rw----. 1 mysql mysql 79691776 Jul 30 05:46 ibdata1
-rw-r-----. 1 mysql mysql 67108864 Jul 30 05:46 ib_logfile0
-rw-r-----. 1 mysql mysql 67108864 Jul 29 01:10 ib_logfile1
-rw-r-----. 1 mysql mysql 12582912 Jul 30 05:46 ibtmp1
drwx------. 2 mysql mysql    32768 May 23 13:06 moodle
drwx------. 2 mysql mysql     4096 Jul 28 11:13 mysql
drwxr-x---. 2 mysql mysql     8192 Jun  8 14:34 performance_schema
drwxr-x---. 2 mysql mysql     8192 Apr 30 13:14 sys

Note the ownerships/permissions on the 'moodle' directory which contains all the raw files for the moodle DB.

On the crashed server, there should also be a config file ... linux typically
found in /etc/ and called my.cnf ... that's a text file and can be edited.
Acqure that file as well.

https://dba.stackexchange.com/questions/111629/table-doesnt-exist-in-engine-after-moving-tables-from-one-vps-to-another

https://stackoverflow.com/a/11506495/2364405

Know this is painful, but please answer the questions at the top.

'spirit of sharing', Ken

New and older servers are running Ubuntu 18.04

Old database was called moodle, new server database is moodle new. Copied the entire folder contents from the old server into the new one. Will try re-adding the folder contents with sql service stopped.

Database name has been changed in config to match the new database.

Old server crashed essentially due to running out of storage (unknown why at this stage priority is to get moodle back up and running)

Unable to run mysqldump or install phpmyadmin due to disk storage and read only issues. Only managed to get files by booting up into a live CD of ubuntu and access them that way.

Appreciate all your help so far just need to get it sorted as soon as possible as you can imagine.

Thanks for answering questions ...

I have had experience at attempting to recover a crashed server (database/mysql) due to no space.  That was a really terrible experience and recovery of that database failed.  Database was corrupted ... period. 

MySQL wouldn't launch due to no space.   The only saving grace to the situation was an sql dump of the database that was a week old.

This next suggestion might be an exercise in frustration, but it might be worth a shot.  

Shutdown mysql, apache and prevent them from launching.  Consult Ubuntu docs on how to do that.  Once you acquire some free space you can set them up again to launch automagically.

Something is taking up space on the old server.  Now you can't do anything about moodledata/filedir as that would require the DB to be running (meta data), but look in moodledata/temp/backup/ ... you might have failed backups in there that are really of no use any more and you can manually remove them all.  In moodledata/trashdir there could be gigs worth ... manually remove anything in moodledata/trashdir/

On the old server, find other things that might be taking up space ... another area might be the logging that server was doing ... /var/log/ Ubuntu boxen to keep old logs that are rotated.  Logs for apache server would be two ... access.log and error.log.   One might be able to get enough space by manually removing the archived logs leaving only access.log and error.log.   Do the same with other logs ... keeping the active one and removing the archived log files.

Got EMail on that server?   Inboxes might be large ... remove those.

Can't tell you when you'll have enough space to launch MySQL there, but after manually removing un-needed files one could check free space via du -h command .... might be able to find other things that can be removed ... with operating system package manager.  IF, for example, you had unoconv document conversion setup in the moodle, that required full blown libreoffice suite installed.  Could remove with package manager libreoffice.   Yes, just broke moodle, but that could be fixed/addressed later.

Dunno if you have such a thing, but many entities have systems that backup entire servers.  Not daily, but maybe weekly?    If you have that sort of service, might be best to restore the entire server image ... even if it's a week or two old.   That would mean loss of data but it's up.  If you do that, the very first thing you do is get a mysqldump of the moodle DB and archive it ... download it ... or xfer it to that other box.

No simple answer ... lots of work ... no short cuts.

'spirit of sharing', Ken

Appreciate your response.

The old server had nothing else hosted on it apart from Moodle.

Could you talk me through trying to sort the version issue out on my new server?

As you can imagine I am under a lot of pressure to get this issue resolved and in my eyes is to sort the version error out on my new server rather than spending hours clearing space on my old server. The old server also has issues with not booting into the gui which is causing even more frustration.

No such thing as 'walk through' in this case ... if the database was already corrupted on old server when you xferred the raw moodle db files to new server then .... ????? ... me thinks still corrupted.

Sort out the version is exactly what I tried to do ... but you say phpmyadmin has issues reading the DB which indicates it's still corrupted.

Hi Ken

I have now managed to get SSH access back into the old server. First attempt this morning was to install phpmyadmin to look at the database and take a backup but got the following errors

W: Not using locking for read only lock file /var/lib/dpkg/lock

W: Not using locking for read only lock file /var/cache/apt/archives/lock

E: You don’t have enough free space in /var/cache/apt/archives

Thanks again for your help on this I hope we can get it resolved soon.

Jordan

I'll send a PM (private message) to you with another EMail address for me so we can converse off list ... forum is delayed and you need this like yesterday.

But before I do that, now that you have ssh access to server ...

If you already had a space problem why would you attemopt to install
*any* software?  The errors relate to trying to install software AND not having space!

Plus, phpmyadmin requires apache to be running.  We might need as much
memory as we can get for the dump of the database.

Please gather info and respond to the EMail address I send to you via PM.

Let's make sure mysqld isn't running or trying to run.

ps aux |grep mysqld

Even if it's not running right now, it might be trying to launch so.

sudo service mysqld stop

Now, let's get a look of what you do have.
Issue: sudo df -h
(that's disk free).

Need to see it all ... we might have room on the drive for re-locating the databases where there is space.

Here's an **example** of output - *** yours will NOT be the same ***

[root@sos httpd]# df -h
Filesystem                           Size  Used Avail Use% Mounted on
/dev/mapper/cl_sos-root               50G   31G   20G  61% /
devtmpfs                             7.8G     0  7.8G   0% /dev
tmpfs                                7.8G     0  7.8G   0% /dev/shm
tmpfs                                7.8G  353M  7.5G   5% /run
tmpfs                                7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda1                           1014M  304M  711M  30% /boot
/dev/mapper/cl_sos-home              239G  112G  127G  47% /home

And let's get a look at how much space is used by the database files:

cd /var/lib/

du -h mysql

What ever it says ... that's how much space we need ... minimally.
Of course we want more.

We need to make sure you do have all of moodledata.
The path to that could be found in config.php of moodle code.

That is typically located in /var/www/
So from terminal ...
cd /var/www/
ls -d moodledata

Is it there?

Then issue:

du -h ./moodledata

In the following **example** am using a moodle27data directory.

The tail end of that will show *something* like:

0    ./moodle27data/lang
76K    ./moodle27data/environment
0    ./moodle27data/trashdir
4.2G    ./moodle27data

*** do the same commands on the new server where you transfered the moodledata
directory.  If you got it all, the last line should be exactly the same or really/really
close.

Ken

Hi All, getting the below error when completing a test or opening some files on my moodle site.

An error occurred while processing your responses (Error writing to database). Click continue to return to the page you were on and try again.

Link takes to blank forum post.

Can navigate and make changes to profile (example) fine

When to run http://serverip/admin/cron.php and get the below error:

Server Time: Thu, 02 Aug 2018 19:24:05 +0100


Execute scheduled task: Site registration (core\task\registration_cron_task)
... started 19:24:05. Current memory use 3.5MB.
Your site is not yet registered.
... used 1 dbqueries
... used 0.014724016189575 seconds
Scheduled task complete: Site registration (core\task\registration_cron_task)
Execute scheduled task: Cleanup old sessions (core\task\session_cleanup_task)
... started 19:24:05. Current memory use 4MB.
... used 62 dbqueries
... used 0.15144801139832 seconds
Scheduled task complete: Cleanup old sessions (core\task\session_cleanup_task)
Execute scheduled task: Delete unconfirmed users (core\task\delete_unconfirmed_users_task)
... started 19:24:05. Current memory use 4.1MB.
... used 1 dbqueries
... used 0.00056099891662598 seconds
Scheduled task complete: Delete unconfirmed users (core\task\delete_unconfirmed_users_task)
Execute scheduled task: Delete incomplete users (core\task\delete_incomplete_users_task)
... started 19:24:05. Current memory use 4.1MB.
... used 0 dbqueries
... used 5.5074691772461E-5 seconds
Scheduled task complete: Delete incomplete users (core\task\delete_incomplete_users_task)
Execute scheduled task: Clean backup tables and logs (core\task\backup_cleanup_task)
... started 19:24:05. Current memory use 4.1MB.
... used 2 dbqueries
... used 0.031139850616455 seconds
Scheduled task complete: Clean backup tables and logs (core\task\backup_cleanup_task)
Execute scheduled task: Background processing for tags (core\task\tag_cron_task)
... started 19:24:05. Current memory use 4.1MB.
... used 6 dbqueries
... used 0.073811054229736 seconds
Scheduled task complete: Background processing for tags (core\task\tag_cron_task)
Execute scheduled task: Cleanup contexts (core\task\context_cleanup_task)
... started 19:24:05. Current memory use 4.1MB.
 Cleaned up context instances
... used 6 dbqueries
... used 0.0044808387756348 seconds
Scheduled task failed: Cleanup contexts (core\task\context_cleanup_task),Error writing to database
!!! Error writing to database !!!

Anyone reading this - I  would high recommend Ken. He has gone above and beyond in his efforts in resolving my issues.

Hi all,

Can you backup posted announcements and then restore them into a new course?  I create about 40 announcements each time I teach and would like to not have to recreate these announcements every time.  Help!

Hi

Does anyone know if can make a backup of users with them profile images and if so how is it run?

Hi Juan,

You could create a backup of posted announcements by backing up the forum and including user data. However, when you restore the backup file, all users enrolled in the original course would be enrolled in the new course! Thus, you should create the announcements in a course with no other enrolled users.

An alternative to using backup and restore functionality would be to keep your original course with announcements forum hidden, and then copy and paste each announcement when you need it.

Evening All,

Got a bit of an issue arising with backing up and restoring from one moodle to another.

I have roughly a couple hundred or so courses to go from one moodle to another, these are all used by users throughout the institute and make use of MNET to log in users using network accounts, multiple users are enrolled on each course both as Teachers and students.

While restoring backups that contains no users (guest accessible courses) the restores fails due to incorrect role mapping.

The two issues are:

array(1) {

}

Looking in to the backup mbz file, under the roles.xml file I can see that it does appear to define the roles

<role id="7">

</role>

</role>

Is this an error that it is trying to take across the roles which cannot be defined by anyone, not even the Site Admin accounts?

If I do this manually, I am presented an option to restore the role mappings by selecting another role to use but I have been automating this with Moosh which does not present the option, failing instead.

Alec