Oh, I noticed a pattern in the output of the $this object that makes me suspicious.
I lack an output of $this when the system is functioning properly to know if this is an issue or not but there are a lot of entries in the output reading *RECURSION*. They are scattered throughout.
Am not a coder. ;)
Files in moodledata/temp/backup/ can be manually removed and no harm will come to the system ... Linux or Windows.
Before you do, however, do look at those .log files that are larger than 0 bytes in size for clues. You found one, BTW, the successful completion of a restore ends with the staging getting to 1000 - everytime. You saw one that reached 700 ... incomplete.
Now the question becomes why/what? That I can't really tell you ... many times course dependent.
You say it happens immediately ... so it never gets a chance to 'build a plan' and then execute that plan ... reporting reaching stage numbers. So what else might be going on at the time. Are you attempting during prime time usage by students/teachers? Got anything doing a virus scan or backup of something?
Check your web service logs and db server logs. If running MySQL/Apache, one might see a "MySQL server has gone away' error. Need to increase max_allowed_packet. LIke I may have mentioned before ... courses tend to get larger ... not smaller ... thus require more resources of the server.
Backing up using the backup.php script in code/admin/cli/ is a good idea .... make sure you use the option to save the backup to a directory outside of moodledata/filedir/ ... so you can find it later by a humanly recognizable name.
continuing to share in ... 'spirit of sharing', Ken
I'm uhh.. definitely a "coder". Extensive development history. I'm okay with taking on the task here, I'll keep digging. I have one month. I have to have a cure before next term. We create those courses (with my software of course) by backing up original templated course, then restore into the new term courses, one for every time slot we teach that course in. That mechanism is the only way we can make this school run efficiently. It would be a zoo gone wild, without it. It's a CLI script. So I'm going to write a companion script to test backup that way. Perhaps there will be a different behavior than interactively.
Lucky for me, with PHP you can output anywhere in the code using echo command. I find this is a huge huge tool for solving problems. echo "hello world"; or, better yet echo "<h1>hello world</h1><br />"; if you really want to see it. Anyway, back to the problem at hand.
So I looked in the temp folder and found one of those logs that had text in it. The rest didn't. Perhaps this is a clue. I've included that log in this reply.
I tried to backup using the CLI approach and ran into the same problem. This time it is very easy for me to find the actual backup folder and log because I had deleted everything. So, I have included the folder and the empty log file in a zip file in this message if that is useful in this process.
Having had a look at the contents of the folder, I see that it seemed to correctly create an xml file for the course (see /course/course.xml file) but it did not finish making the /course/enrolments.xml file as it is empty. So perhaps this is a clue if it stopped at that point between making the course and the enrolments files.
Interestingly now that I'm looking at the folder again on the server, I see that Moodle has graciously created about 40 other folders for backing up. So there must be some mechanism that is going haywire behind the scenes and still executing somehow.
Happen to be working with a Moodler who does same process ... backup no user, restore to new site.
On his site/hosting, mod_security is installed and active.
The error you are seeing indicates, obviously, cannot finish ... something interrupted the process ... not serious enough to crash Moodle, but serious enough to either kill a backup or complete it with missing parts/pieces. But what? I know not!
There is a command line utility (swiss army knife type like that of drush for Drupal sites) called moosh.
Works well ... on Linux.
The file have attached indicates that restore was successful ... it reached stage 1000.
There shouldn't be a 'matching' directory by "b26146b9868f832de5b5fdef60c81395" in there ... the last two actions is to ***copy*** (note that is copy NOT move) the backup file to configured destination directory ***AND*** then destroy the temp directory being used to build the backup file.
So, sorry, no clue in that .log file. As I tried to explain, the log file means something to coders ... hard core coders ... but not to me ... I asked one time in these forums what the numbers meant and if they were tied to a process for a part/piece of making the backup file. Don't recall getting any response, but then again, if there was, I might not have understood it.
It's only by observing the process that I've concluded that a stage ID number IS associated with making a backup of oh say 'activites and resources' (mods) ... you will see an xml file ... road map to activities directory. A files.xml ... road map to files in files directory but not by humanly recognizable names .. but, rather, built the same way files in moodledata/filedir are stored.
Now if you've created your own program, suggest maybe comparing what you've done to what Moosh does.
Too bad moodle doesn't have a 'restore.php' script in admin/cli/ that works like backup.php.
But, that's where moosh comes in.
There are 44 contributors to moosh
https://github.com/tmuras/moosh/graphs/contributors
Maybe by reaching out to one of them ... the lead ... you could shave off hours and hours ... and get this done!
'spirit of sharing', Ken
Running Windows right?
Is your php 64 bit or 32 bit? You mention what version of Moodle, right?
The zip file - looks like it created an enrollments.xml ... but couldn't populate - 0 byte .xml file.
That's a clue ... enrollments on site/courses are handled how?
'spirit of sharing', Ken
Turn autobackups off.
Manually remove all the contents of moodledata/temp/backup/
After that, do what many Windows server admins do ... reboot the box.
Check the reports for autobackups in Moodle admin. You might have to truncate all tables moodle keeps related to autobackups to 're-gain control' .. perception of 'control', I should say.
'spirit of sharing', Ken
Hi Ken,
Autobackups have never been allowed. It's always been off.
Strange thing is now that I've returned back from lunch there's a whole bunch more folders and logs created. What other mechanisms would initiate backups aside from the Autobackups and someone doing it interactively on the website and of course, my own CLI scripts? Are there any other mechanisms that would perform a backup?
One thing I have discovered is that if I perform a backup of a single activity by first clicking on the activity and using the Backup menu option for that specific activity, the backup performs accurately without errors. I'm not exactly sure if I can include more than one activity but I'm about to begin trying various partial backups to see if I can whittle down to the offending activity, block, or whatever.
Thanks for sharing your time with me.
Never allowed means what? Never turned on?
Do you have more than one admin level user?
Has autobackups been turned on without your knowlege?
Other than teachers themselves doing backups, in Moodle code
only the cli script in moodlecode/admin/cli/autobackup.php
provides that function. And to access/run those one has to be logged on and execute via
command line. Unless you've hacked code.
Obvioiusly, factors ... your script/code. You've not shared any info
on that. Have never heard anyone say system doing backups all by itself.
Did you look at reports for automated backups in Moodle Admin? Although nothing would be there on something you've programmed me thinks.
This eror "cannot_finish_ui_if_not_setting_ui" means something interputing a backup process. Seen that in a situation where admin was attempting to restore the same course multiple times by using same browser with 3 active tabs attempt to perfom a restore of the same backup file. That system (a 3.5.3+) was loosing track of stages etc. Errors always showed staging had reached 700.
Now what about your scripts? How are they run? In some sort of timed process? What happens when one fails ... does it log which backup failed and schedule that course fora later time?
Am not capable of Vulcan Mind Meld! :| Encourage you revisit your scripts/system/logs, etc.
'spirit of sharing', Ken
Hi Ken I thought I was clear. Never on means it has never been turned on, never allowed, never used. I found what is doing the backups though and that is adhoc tasks. Apparently if someone makes a modification to a course that was created using a restore (I think that's what triggers this along but not sure, is if it was a restored course) it makes an entry in the adhoc system and then the "cron" job executes it until it's finished.
I found this verbiage in one of my "cron" job logs.
Execute adhoc task: core_course\task\course_delete_modules
... started 13:30:06. Current memory use 33.7MB.
<h2>got here! backup_aclaurus_activity_task.class.php</h2><br /><h2>got here! backup_aclaurus_stepslib.php</h2><br />... used 196 dbqueries
... used 2.0592029094696 seconds
Adhoc task complete: core_course\task\course_delete_modules
The <h2>got here... portion that is what I put in my module code to let me see debug information when a backup and/or a restore is performed on a course with the aclaurus module in it. So that's why I see it here. So I am very puzzled, and I see very little documentation for this. Why does a backup need to be performed?
So now that I know what is causing the backups to be created I can rest that part of my mind worried about if the system has a rogue process. It doesn't. But I am very perplexed why a backup would be triggered. What does that backup serve? There is no automatic backup being done. Where does this backup go to (considering that it finishes without errors, which it does not at present time).
I'm going to turn my direction back to the error in the system. I must understand WHERE the problem is thrown.
I believe it is a module that has a bug in it. I tried to rule out my own module(s) first since I have done nothing to upgrade them in the last site upgrade, but honestly, none of my modules have anything special in the backup and restore sections of the modules. They are all using "stock" code for the most part gleened from examples here on moodle.org development documentation.
Also I have confirmed that none of my modules throws an exception. The problem happens before any of my modules backup code gets fired.
Getting closer ... first ... ad_hoc cron, if I re-call, is an old cron process. Don't re-call in which version of Moodle but there is now a recycle-bin for a course. Deleting module in a course would trigger the re-cyclebin for the course which uses backup to keep the resource/module in a position to be restored.
Best guess!
'spirit of sharing', Ken
Have you looked at your plugins? Try disabling all addons and then run it...
Turn full debugging on and run and then see if you get additional error messages...
I found the problem! Yay! Now how to fix it?
I went deep. I isolated the problem making a new script and adding all kinds of echo commands. The problem is I have a table missing in our database. How it is missing is beyond me, but I need to create a table named mdl_enrol_lti_tools.
I discovered this by outputting the exception object using print_r(). Then in that data, I saw the error clearly...
EXCEPTION!!!!
dml_read_exception Object
(
[error] => Table 'moodle.mdl_enrol_lti_tools' doesn't exist
[sql] => SELECT * FROM mdl_enrol_lti_tools WHERE enrolid = ?
[params] => Array
(
[0] => 12147
)
[errorcode] => dmlreadexception
[module] => error
[a] =>
[link] =>
[debuginfo] => Table 'moodle.mdl_enrol_lti_tools' doesn't exist
SELECT * FROM mdl_enrol_lti_tools WHERE enrolid = ?
[array (
0 => '12147',
)] snipped here for brevity.....
So, now it's a simple matter to find the definition for the table and create it manually and I believe the backup should work.
It works! I had to create more than the one table. I had to create two tables.
Both were missing. How could this happen? Very strange. Well, it's fixed. I got the definitions from XMLDB and recreated manually and now everything is fine.
Congrats! Indeed ... what happened to those tables? Who knows!
In looking over this thread, at the beginning, you said ...
"We create those courses (with my software of course) by backing up original templated course, then restore into the new term courses, one for every time slot we teach that course in."
Have just finished helping a gentleman do pretty much the same thing with scripts
from command line. Using moosh. OP made one no user backup of his
template course (no user, with whatever theme/addons/blocks, etc.). Saved the backup to a location on the server and renamed it 'restorethisone.mbz'.
In the admin/cli/ area of one of the moodle instances, this bash shell script (called restoretemplate here):
(contents below)
echo 'Template course restored to Misc. category of site. Cat ID 1';
moosh -n course-restore /path/to/restorethisone.mbz 1
Restores the template backup to category ID 1 which is Miscellanous by default. Pretty easy for Op ... just executes the script once ... like ./restoretemplate [ENTER]. When it finishes, 'up arrow' for command line repeat previous command, ENTER ... repeat ... repeat ... repeat ... however many template courses OP desires to restore that session.
Template courses are restored like Template, Template copy 1, Template copy 2, etc.
however many times the op executes that script.
Op then logs onto the site, accesses each course and changes basic info in a course to 'band them'. Template copy 1 might become "Puncuation and Grammer 101 - SMU", Template copy 2 might become "Puncuation and Grammer 101 - Texas Tech", etc. and then moved to an appropriate category via Moodle UI.
And Op ends up with the same thing ... only no programming required.
'spirit of sharing', Ken
That sounds like a lot of work to me! To be honest. Wow. I have a fully automated system here. I rarely have to do anything regarding new class creations and backups and tons of other stuff, like attendance. We have Registrar staffing that takes care of classroom assignments with our entire student body. We have a student management system, hybridized from a commercial product and things I've added in to accommodate our needs with a web interface I made also. That system is what they use exclusively. The programs I wrote to bridge Moodle with the backend system were a bit of programming, sure, but that time has paid for itself so many times over. Without those programs and custom software they would need to hire two more people. And on top of that, I'm free to use my creative skills in other areas for the college that they didn't hire me for, which is what I've been doing for years.
I also made a "Web Lab Server" for our Web Design program. Students in that program have their own webspace, with PHP, MySql, ASP.net 4.5 support. Each one gets their own webspace with their own path. And that system is automated too now. New enrollees are automatically created in the Web Lab system. This allows them to have a public place on the internet to show off their work, and/or utilize for classroom work, homework, etc.
Anyway, there'd be no reason to go back to a system of scripts that require a human being to execute many times over, changing file names, copying files, etc. However, coming from a place of nothing and doing everything manually otherwise, your approach makes logical sense.
Thanks for your input along the way. I'm very pleased I found the cure. To be honest I was doubting if Moodle was still using the same Backup 2.0 object methods. Glad to know it was a database issue. Now, on to discover how those tables went missing...
Hi Perry,
great investigative work!
If I were you I'd file an issue - maybe w/ low priority - into the Tracker since tracing that exception in the log would be much better, regardless your issue was having a "broken system" from a DB perspective - wondering how LTIactivities could work when accessed.
TIA,
Matteo
"A lot of work' ... well considering your resources (multiple people doing jobs related) then yes, my 'sharing' might seem ancient ... for a single OP who wears all hats in the entire setup of his system - includes Paypal, etc. could have expanded that simple script to do a lot more + setup a Webmin that would provide much more 'automation' as well. Single OP ... mind ya ... wears all hats.
But ... more power to ya ... you need all the automation you can muster sounds like.
Suggest filling out your profile with a description of your system(s). Other folks would be interested!
Good luck finding out why those tables were missing!
'spirit of sharing', Ken
I'm really not sure what you mean by "store it as Package"
- Moodle will backup courses in a .zip format, with a .mbz file suffix.
Site backups are one thing - and you could probably do this just by creating snapshots of your server. If you want to take a copy of a whole site, everything, and restore that (as a new site?) then server snapshots is the way to go.
However, when you talk about backing up courses, then all activities within a course will be backed up within each course - e.g. the Wiki that you refer to.
Hi Stefan,
Sorry i don't provide a direct solution for you.
But perhaps you can vote/comment for MDL-52050, that could finally help for your need.
You can also create a little bash script, calling in a loop (with all needed course IDs) the /admin/cli/backup.php script. But it will use your current settings for backups (so you need to modify them before and after).
Séverin