
I think you nailed all the main points about why a lot of games in that generation were so difficult. I always think of the arcade machine connection; that designers were still stuck thinking they had to make games tough to keep you putting in quarters, even though they didn't have to anymore.
I'm trying to think of difficult games in the Atari generation, but I can't really think of any. Not like Ninja Gaiden anyway.
It would be interesting to read a book like Racing the Beam (about programming for the Atari 2600) for the NES or Genesis.