by Colin Fraser.
Then piecemeal is perhaps the only way this will work. I would probably be using phpMyAdmin if you are using MySQL for your database, or phpPGAdmin (I think it is) if using PostGreSQL. You have more control over the data selection and end product - what records you want to keep or lose, essentially.
The alternative is to go into the php.ini file and make sure that you have allowed sufficient resources for a large backup to work.
EDIT: I just remembered you talk about a "choked up" database, you mean one overly full for the data it contains? IS there an issue inside the database, duplications of something? Better check that out.