I never found the 2-D versions of FF4 to be very difficult, although it was still somewhat more difficult than VI (which is far and away the "easiest" FF game). The DS version, however, is quite brutal.
I'd say the toughest of the main FF games is probably FFV. In that game, the difference in power between your party members and the enemies is very thin, and the bosses are cheap bastards.