Quantcast
Channel: Backup and restore
Viewing all 6640 articles
Browse latest View live
↧

Automated Course Backups Failing Randomly

August 31, 2015, 5:22 am
$
0
0
by Joshua Bragg.  

On 3 of the past 4 nights, one of our automated course backups has failed.  It has happened on two different courses (one twice, the other once).  The error message has been the same sort of thing each time:

Exception: directory_not_exists /var/www/vhosts/rjreynolds.org/moodledata/temp/backup/9003b2869d9194ccf31ac22c4aa8e3d3/activities/quiz_7290

The googling I've done so far shows some old problems but I'm wondering if this is related to the changes in MDL-32547.  We've never had backups like this failing and it has started only since we upgraded to 2.9 over the summer and courses opened up last week.

Thoughts?

↧
Search

Re: Automated Course Backups Failing Randomly

August 31, 2015, 8:54 am
$
0
0
by Ken Task.  

Not sure the tracker items linked and followed are the same thing.

This sort of thing requires some sluething, me thinks ... soooooooo ...

IF one were to inspect the questions.xml file in the failed directory (that long one in your error

9003b2869d9194ccf31ac22c4aa8e3d3)

will see a contextinstanceid tag.  Like (example):

<contextinstanceid>94</contextinstanceid>

There needs to be a corresponding directory in activities:
quiz_94 (as per example)

That directory is missing in your issue ... quiz_7290.

Search questions.xml for 7290 to see if there is an entry.
To find the question in the course:
questions.xml will have a name tag
Example:
<name>Google Applications Quiz</name>

Go to the course and check out that quiz.

Might also run a health check (development/not supported) ...
As admin level user via
http://site/admin/tool/health/

It might report issues with quizzes and offer suggestions as to how to fix
(via sql queries).

'spirit of sharing', Ken

↧
↧

Re: Automated Course Backups Failing Randomly

August 31, 2015, 12:17 pm
$
0
0
by Joshua Bragg.  

There isn't a question.xml file in the backup folder.  I'm  guessing this is because the backup didn't finish.  There are also no quiz folders in there after quiz 7290 either even though there are probably 15 quizzes or so after it in the course (the quiz ID numbers are all consecutive because they were restored from a backup in June.)  The quiz is perfectly functional on the site.  The far easier way to do this is just to visit site/moodle/mod/quiz/view.php?id=7290 and see what it is.

I should have included this also.  It is my personal course that didn't finish backing up twice.  Each time, the quiz ID that the error message provides is different.

Thanks for the pointer to the health check script.  I've missed that one somehow.  I ran the health check and did get some issues with "Question categories should belong to a valid context" and "Question categories tree structure".  However, from all the ones I've spot checked so far in the database, they look like very old cruddy data that isn't connected to either of the two current courses I've had errors on.  Worth looking into more deeply but it doesn't seem to be connected at the moment.  Our site started off on 1.9.13ish with an export of a course (my course) from a 1.9.3 that hadn't been upgraded ever...  I've always assumed there was some bad data in the database but now I know where to look.

I'll pick my way through the bad data to see if I can clear things up a bit.

↧

Re: Automated Course Backups Failing Randomly

August 31, 2015, 2:22 pm
$
0
0
by Ken Task.  

One of the heaviest processes in anything Moodle is quiz.   What, if anything, do apache error logs say upon the failure of backing up your course manually?

No questions.xml ... is not good ... think Moodle generates .xml files first to use as road maps for each part/piece of the backup process.   Are we running out of memory?  The default PHP limit of 128Meg for a 2.9.x Moodle is too low.   When investigating how 2.9 does backups, had to increase not only the memory allowed to use but also, the time for a script to run - and that was on a small 1 quiz course.

Can see those values in Server's PHP Info.

If those look acceptable, then ...

Ok, know what I'm about to suggest is a pain ... but if no question.xml was present ...

2.9 has that 'easy skip to end' button so users don't have to step through the various backup screens.   Step through the screens ...

Here comes the pain ... create a no user backup and exclude all quizzes but do include all other activities.   Name the backup with a -nq (no quiz) in it's name - no users selected will auto insert a 'nu' at the end of the backup file name.   Did it finish?

Then create another backup this time selecting no activities but the first half of the quizzes.  Filename: backupblah-qz-1-10

Did that one finish?

Then create another backup this time selecting  no activities including the first half of the quizzes.  Filename: backup-blah-qz-11-20.

Did that one finish?

There's something else to try ... command line backup of the course ...

cd moodlecode/admin/cli/

php backup.php --courseid=[courseIDnumber] --destination=/pathto/some/large/space/you/can/access/

That takes apache out of the loop ... just php and mysql then.

'spirit of sharing', Ken


↧

Re: Automated Course Backups Failing Randomly

August 31, 2015, 7:30 pm
$
0
0
by Joshua Bragg.  

I think we're talking about different things on some of this.

To be clear about a few things:

  1. This only happens for two courses so far but it is not an every night thing.  My course for example failed to backup Monday morning and Friday morning of the last week but backed up perfectly every other time.
  2. I have already done full backups of both courses manually in the past and they complete successfully.  I'll try to run one of the beefy backups again later this weekend when the server is quiet just to double check.
  3. When I say the question.xml file is missing from the backup folder I mean that its missing from the moodledata/temp/backup/###hashgibberish###/ folder and subfolders.  That's not surprising to me since the backup didn't complete.  I didn't need to look into one of the completed backups to figure out which quiz it stopped on because I could just use the URL trick to see.  I should have been more explicit here.

Thanks for all your suggestions.  I'm just trying to make sure we're talking about the same thing. 

↧
↧

Re: Automated Course Backups Failing Randomly

August 31, 2015, 9:28 pm
$
0
0
by Ken Task.  

So you are doing automated backups every day ... in the morning?  I hope wee hours of the morning and not prime time for usage of other courses/system.   Failed only  on Monday and on Friday.

We've not mention specific system details ... maybe it's time ...  shared system?  How many other host on it?  Is the shared system a cheapo package where the provider has strict limits on processing power, space, data throughput, etc. on each customer?   True dedicated server or a VPS?  On a VM?

but if that is a pattern ... failure on Monday and Friday ... that sorta points to something happening at that time that sucks up all resources and causes the failure to occur ... a restart of something which kills the cron job backup process?   Database on same server or is it a remote DB server?

So what else do you have scheduled in Cron?  

Is this server in a Virtualized environment?

And there are no hints in error logs of server ... apache error log, php error log, secure log, whatever daemon running log?

Had a server in a Virtual environment once that for 5 minutes of every hour at exactly 32-37 after the hour would become un-responsive ... server up, apache running, mysql running, but no one could access.  Finally got the virtual server administrator to look into it.   Problem solved ... nothing wrong with the guest OS upon which Moodle server was running ... it was some other guest OS on the same physical Virt server that had been granted use of all resources every hour at 32-37 after the hour to do whatever it was doing.

There's just too many folks that do run autobackups and don't have issues (yet).   At least none like your description, so without live monitoring of the system at around the times the autobackups are to be run, just about anything we talk about/suggest is a guess. :|  Lot's of what appear to be un-related questions and maybe side trips (inspection of the remaining temp directories, for example).

'spirit of sharing', Ken


↧

Re: Automated Course Backups Failing Randomly

August 31, 2015, 11:11 pm
$
0
0
by Tim Hunt.  

Well done for finding that bug Joshua. That could well be the cause.

Have you managed to establish an exact time-line for what is happening? That might give some clues, but since you don't want to sit up all night, it would probably need to be re-constructed from log files. When cron runs, it outputs quite detailed log information, but it is not very standard where/if that it saved. Can you track it down?

The automated clean-up of 'old' backup data does have a config setting configtempdatafoldercleanup. Can you try making that longer, and see if that 'solves' the problem. Oh, I see. It seems to default to 7 days (the longest). That does not seem like a problem.

Very odd.


↧

Restored courses aren't overwriting old

September 1, 2015, 12:34 am
$
0
0
by Leticia Dark-rose.  

Hi,

I am trying to refresh 2 courses currently found on our UAT site with the updated courses from DEV. I have been able to backup the course fine and restore it into UAT, (no errors) but when i look at the course it hasn't been updated.

Does anyone have any ideas? We are running Moodle 2.8

Much appreciated.

Leticia

↧
Search

Re: Automated Course Backups Failing Randomly

September 1, 2015, 3:47 am
$
0
0
by Joshua Bragg.  

Tim, your comment about timelines put two things together in my brain that I looked at recently.

Last night, the automated course backups started at 12:50AM.  It proceeded rapidly through 8 or so courses finishing them by 12:53AM.  My course started backing up at 12:53 and failed at 12:56:01.  It then proceeded through the rest of the courses and finished by 12:58.  Now its pretty typical that my course takes the longest since I have an absurd number of questions and quizzes.

The part that is interesting here is that "Delete stale temp files" task ran last night at 12:56.  In fact, all 4 of the automated backups that have failed in the last week all failed at either 12:56 or 12:57.  This task is scheduled to start on the 55th minute every 6 hours.

So I should just be able to edit the schedule of the automated backups and delete stale temp files tasks and avoid the collision.  With that said, I won't do that for now so that I can help with debugging.  I'll see if I can track down where/if cron is being logged on our system.  I have a feeling its going to /dev/null though. wink

↧
↧

Re: Automated Course Backups Failing Randomly

September 1, 2015, 3:55 am
$
0
0
by Joshua Bragg.  

Yes I am running them early in the morning.  In fact, one of the full site backups I run disables access to the site.  Every now and then I have a kid complain that they were working on something at 3AM when that happens and I get to give the poor high school student who asked a long explanation about proper sleep habits.  smile

This is a VM and I'm sure there are lots of other sites on the box, but the box isn't mine so I can't tell you.  I'm allocated a specific amount of hardware.

I'm going to see if I can track down some cron logging (or create it) after my epiphany a couple replies below.

↧

Re: Automated Course Backups Failing Randomly

September 1, 2015, 2:59 pm
$
0
0
by Tim Hunt.  

It would be worth looking at the time-stamps of the files that Moodle is deleting. Moodle should only be deleting files that are more than 7 days old, or something. How old are those files really? How old does Moodle think they are? Is that because the get the wrong time-stamp when they are written, or do they have the write time-stamp, but somehow Moodle does not see that?

↧

Re: Automated Course Backups Failing Randomly

September 1, 2015, 6:02 pm
$
0
0
by Joshua Bragg.  

I looked at the most recent failed backup in the temp folder and all the time stamps seem to be in line (Sept 1 between 00:53 and 00:56).

I've got the cron logging set up to happen overnight while the tasks run into each other.  We'll see what it looks like in the morning.

↧

URL links in a restored course

September 2, 2015, 1:11 am
$
0
0
by Tanja Rykovska.  

Hello, I have restored a course (that I created during a training) on Moodle cloud and it seems that URL links don't work - there is a frame for an embedded link but the content of the website is not displayed. What has happened and what can I do about it? Thank you! Tanja

↧
↧

Re: Automated Course Backups Failing Randomly

September 2, 2015, 3:38 am
$
0
0
by Joshua Bragg.  

Everything went well last night.  I'll keep an eye on things and let you know if I see anything else.  There was a course that overlapped with the stale temp files task but it didn't seem to cause problems.

↧

Upload Courses feature & Repositories

September 2, 2015, 2:54 pm
$
0
0
by Justin Almeida.  

When uploading courses via .csv file, one of the options is to specify an absolute path to the backup file.


backupfileAn absolute path to a backup file (.mbz) to import in the course.


My question is, do repositories have an absolute path?

I did do a bit of searching, but was unable to locate an answer.

Thanks!

↧
Search

Re: Automated Course Backups Failing Randomly

September 3, 2015, 4:19 am
$
0
0
by Joshua Bragg.  

The error happened again last night, at about the same time.  I've attached a section of the cron log between 00:49 and 1:00 when all the courses have finished backing up.

Here is the relevant snippet:

Server Time: Thu, 03 Sep 2015 00:57:01 -0400

Execute scheduled task: Delete stale temp files
... started 00:57:02. Current memory use 18.4MB.
backup_auto_failed_on_course DemonChem
Exception: directory_not_exists /var/www/vhosts/rjreynolds.org/moodledata/temp/backup/25d3b4b9248732c758fca0aa6588b48c/activities/quiz_7194
complete - next execution: Fri, 04 Sep 2015 00:30:00 -0400
Backing up e-Commerce I (Blair)...
... used 0 dbqueries
... used 2.2312309741974 seconds
Scheduled task complete: Delete stale temp files
Execute scheduled task: Cleanup files in trash
... started 00:57:04. Current memory use 19.5MB.
Deleting old draft files... ... started 00:57:04. Current memory use 21.8MB.
done.
Deleting orphaned preview files... ... started 00:57:06. Current memory use 21.9MB.
done.
... used 11 dbqueries
... used 2.2669699192047 seconds
Scheduled task complete: Cleanup files in trash


I must admit that these logs are hard to look at because different parts seemed to be mashed together sometimes based on when the various tasks complete.

There are also a few of these errors in the log at various points.  They start happening during the time that the automated backups are running and don't seem to happen outside that window.  Here is that snippet:

Default exception handler: The operation timed out while waiting for a lock. Debug:
Error code: locktimeout
* line 446 of /lib/classes/task/manager.php: moodle_exception thrown
* line 66 of /lib/cronlib.php: call to core\task\manager::get_next_scheduled_task()
* line 61 of /admin/cli/cron.php: call to cron_run()
!!! The operation timed out while waiting for a lock. !!!


I'm now in officially over my head as far as interpreting these things so I'll appreciate whatever you have for me.  If it would be helpful to you, I do have the log saved from last night when all backups completed successfully as well as a much longer log file that I pulled this piece out of.

Thanks all.

↧

Re: Automated Course Backups Failing Randomly

September 3, 2015, 7:33 am
$
0
0
by Tim Hunt.  

I think this has got to the point where you should create a tracker issue with the information you have, and link to this forum thread.

Possibly the only way to debug it more from here is to put some extra debug code into the delete temp files task, to print the relevant time-stamps of every file it considers, and whether it then deletes it or not. However, I don't know if that is within your level PHP programming skills. Perhaps someone could give you a chunk of code to copy and paste, but that person is not me right now.

↧
↧

internal server error 500 when restoring courses

September 3, 2015, 1:41 pm
$
0
0
by Manfred Knabl.  

When I try to restore a Moodle 2.6.1 (Ubuntu, Apache, MySQL) course in Moodle 2.9 (IIS und MSSQL) then I allways get a internal server error 500 after a long time of waiting, although the filesizes are only about 10 M.
I also used proper values in the php.ini for post_max_size, max_upload_max_filesize and max_execution_time, but they didn't solve the problem.

Is it possible to restore a Moodle 2.6 course in Moodle 2.9 or not?

thx Manfred


↧

Re: Automated Course Backups Failing Randomly

September 3, 2015, 9:20 pm
$
0
0
by Joshua Bragg.  

Tracker issue filed: MDL-51323


Sadly my PHP skills are just next to non-existent.  Working towards that slowly.  I mostly finished CSS this summer with Codecademy.  JavaScript basics are next and then PHP.  I keep feeling the need to make myself more useful around here than just reporting bugs.  

↧

Re: Moodle 2.9: Restore - Error connecting to the server

September 4, 2015, 3:10 am
$
0
0
by Graham Moir.  

Ken, thanks for taking the time to reply in such a detailed way. Yes, this is a shared hosting environment, and yes I'm restoring backup created on that server to the same server - but different instances of Moodle.   

I haven't taken the route of increasing timeout limits etc.  because your other suggestion regarding the file repository gave me the workaround I needed.  I downloaded the backup and manually FTPed it to the other Moodle repository folder and then restored from there without problems. 

I take your point about just looking for the backup file and avoiding download/upload, but all is sorted now.

thanks !

↧
Viewing all 6640 articles
Browse latest View live
Search
<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>