Have you looked at the error messages?
Go to Site Administration > Server > Debugging and then to error mesages, a dropdown list box and select ALL: Show all reasonable PHP error messages.
Do the backup again, and see what error messages you get, if any. If you get none, then it may be a server error, so check your server logs, or there is something in the course that is not allowing you to back it up all at once?
One thing you can do is break the course backup into two or three parts, by backing up just a part of the course at a time. See if the glitch causing the problem can be isolated. Each backup should have a different name, and when restored, use the "Restore into existing course". That, I have found sometimes, can be more revealing, and less stressfull. Student enrollments go into the first part and not included after that.
If a part does not backup, does the same thing, then just keep working through it element by element, until you are successful. This allows you to you locate, specifically, the element causing the problem.