I can't quite decide which of the two is more nightmarish.
Definitely the whale I'd say. After all, Eig with a tail is sort of a Nessie parody…
Your use of the term "nightmare" makes me think about the role of these creatures in the world though. If at this stage of development, Polokus was just a (the?) magician, then the idea of a benevolent creator-god who dreamed the world into existence may not have been part of the story yet. So then it makes sense for the sea to be full of hostile creatures, since there's no reason for them to be confined to a "Cave of Bad Dreams". In the final game, the sea still contains Piranhas, of course, but they're pretty small and easy to deal with – one gets the impression that the worst infestations were catalysed by the Pirates' pollution. Then the larger and more intelligent aquatic creatures are friendly (Sssam, Carmen) or at worst neutral observers (Eig).
Thinking about the first game, we only saw small fish in the sea, and some were friendly, and others evil, so that's not really much different from what we got in the second game. On the other hand, Rayman couldn't swim, so water was still pretty forbidding. Imagine in this early iteration of Rayman 2, he learns to swim, but then discovers that the waters are full of hostile creatures – that would be pretty tense!
So then, without a benevolent creator-god, and a sea much colder / more hostile than the one presented in the final game (Sea of Lums in the manual, Whale Bay in the game), would that have made Rayman 2 darker? It's funny,
L'histoire de Rayman claims that Rayman 2 was going to be goofy until Tonic Trouble came along, so they made it more serious to differentiate the two. Well, I guess Origins is goofy, but it still shows a sea full of hostile creatures (cf. my "paranoid" sig!).