It turns out I only had a few duplicates. The questions were all in a non-existent parent category which was why they were not visible anywhere in the webinterface. Changed the parent category and then was able to see questions in question bank. Went through and deleted them all in the web interface, which effectively just hides them. However, after doing that, I was able to restore several more courses, and even one with system level questions.
Each course is taking about 1-2 hours which seems ridiculous. Why so long? Never used to take this long.
The second course I tried that uses the same questions as the first errored out with the same error "Did you forget..." as above.
Off to the quiz forum to see if Tim has a script for permanently deleting questions out of the question bank. It would nice to be able to run a script that would find all references and do a clean delete of a question. Seems like this should be an option in the web interface. I am worried that after "deleting" all the system level questions that maybe the next restore created another copy though I have not checked that yet.