Rayman 1 Level Editor/Viewer (Ray1Map)

[PluMGMK]

Looking at the sprites, it seems Joe and the Beach Ball are in the same DES, which I had failed to notice before because I didn't look at exporting the sprites. There doesn't seem to be an ETA for either though, which is very disappointing. The DES is basically a waste of space without its ETA, so if they were concerned about memory I don't see why they would remove one but not the other.

Maybe the Skops boss was originally going to reverse the directions too (at some early phase of development), and the code got muddled up in EDU? That's an intriguing thought, but I don't know if it makes sense in light of what's in the Rayman Classic exe.

[RayCarrot]

Looking at the sprites, it seems Joe and the Beach Ball are in the same DES, which I had failed to notice before because I didn't look at exporting the sprites. There doesn't seem to be an ETA for either though, which is very disappointing. The DES is basically a waste of space without its ETA, so if they were concerned about memory I don't see why they would remove one but not the other.

Oh, that's annoying :/ I wonder why they did that. You could always manually copy over the ETA I suppose, but we can't add it to the sheet then.

Maybe the Skops boss was originally going to reverse the directions too (at some early phase of development), and the code got muddled up in EDU? That's an intriguing thought, but I don't know if it makes sense in light of what's in the Rayman Classic exe.

Not sure. Could also be it was writing some bytes to the wrong location (maybe using a leftover pointer from R1?) and it overwrote the RayEvts flags (which determine Rayman's powers and stuff like the Mr Dark spells etc.).

[PluMGMK]

Per Occam's Razor, I guess that makes more sense. In that case, I guess what I'm seeing is good old fashioned nasal demons Undefined Behaviour. It still doesn't make much sense that Lava is able to cause such a thing though. Actually, it might be related to the way Skops makes the screen scroll, and the lava moves with it. That at least is likely what that enumeration over them in the Skops code is for. Maybe the Lava events themselves are trying to mess with the screen and it doesn't work correctly in EDU. It might be best to make them TYPE_EAU instead to avoid this nonsense, although I'm not sure that that works properly in EDU either (water is suspiciously absent from the Rising Tides level).

On a somewhat unrelated note, the idea of Skops being able to reverse the directions too made me think that he himself might have been the Big Bad early in development. On further reflection that does seem unlikely, but it got me thinking about the fact that the final world was at one stage going to be an Electrical world, before they replaced it with the Cake theme. That may be where the concept of the UFOs came from originally (or at least their wiring and indicators), and might even explain why this was present in Cake. Might be best for me to take this particular discussion to the main Rayman 1 topic.



Speaking of UFOs, I can now confirm that the code still works in Designer, but it expects the event to start in Etat 0 and SubEtat 0, and then moves it into SubEtat 1 when Rayman jumps on it, just like in EDU, and unlike in Rayman 1 (where the SubEtats are 13 and 14). If the event doesn't start in SubEtat 0, then it doesn't work at all. So unfortunately you can't just turn a cloud or pencil sharpener into a UFO. This really feels like one of those things that didn't need to be hard-coded!

Now I understand this hard-coding, I'm starting to get why they reorganized the UFO-related ETAs in EDU. In Rayman 1, the UFO states were on the end of (what was presumably even then called) IDC.ETA, and the lever states were (for some reason) buried in the middle of PNG.ETA, in allfix, and the respective event behaviours were hard-coded to use the particular Etat/SubEtat values therein. This was no longer sufficient in EDU, where they needed completely different sprites for the UFO and lever in Image, while still allowing them to work in Cave. So they broke the lever out into its own ETA/DES pair, presumably called LEV.ETA/LEV.DES, with just two states available, only in Cave. They then created LEV2.ETA/LEV2.DES in Image, with the same states, so the same event code could use both. Ironically, they didn't even use the Cave levers in the final game, but everything's still there to make them work.

Breaking out the UFO itself is a bit less justifiable, since the relevant ETA and DES weren't in AllFix, but obviously they wanted to be able to duplicate IDC.ETA/IDC.DES exactly between Cave and Image. So again, they broke out the UFO into a small ETA/DES pair in Cave, with only two states available, and then created an Image equivalent with a Palette sprite, and no engine sound.

The biggest mystery here is why in Rayman Designer, they cut the ETA/DES containing the Palette, while keeping IDC and LEV2. It makes no sense from a space standpoint, since the palette is literally just a single sprite, while the other two contain several animations.

EDIT: I used the code you posted a few pages ago to port over the Palette to Designer. It feels a bit like cheating, but it works perfectly.

[RayCarrot]

Regarding Skops, could be, but I can't find anything in the code which suggests he could trigger your directions to swap. I did however notice the function RAY_REVERSE_COMMANDS gets called from 3 places; INIT_RAY, DO_DARK_SORT_COLLISION (Mr Dark when he casts a spell on you) & DoAnnuleDarkSortRaymanCollision (the curse cleaner), with it being very odd it's being called from Rayman's init code... That function is sadly not very readable due to how it was compiled, so not really sure when it would get called, but it seems to specifically check for "num_world == 6 && num_level == 3". I'm not sure what they did with all of these level checks and such in the EDU games.

Regarding the UFO, makes sense! It's indeed weird though about them keeping stuff like that in Designer. I also just realized now that some of the .wld files in EDU go above 1.3 GB! I remember a few months back Ryemanni and I tried moving a bunch of DES/ETA between .wld files in R1 and noticed there was a limit where it couldn't be bigger than around 900mb. It seems they increased this greatly in later games. Would be interesting to see how much you can actually fit into a .wld in Designer

[PluMGMK]

Regarding Skops, could be, but I can't find anything in the code which suggests he could trigger your directions to swap. I did however notice the function RAY_REVERSE_COMMANDS gets called from 3 places; INIT_RAY, DO_DARK_SORT_COLLISION (Mr Dark when he casts a spell on you) & DoAnnuleDarkSortRaymanCollision (the curse cleaner), with it being very odd it's being called from Rayman's init code... That function is sadly not very readable due to how it was compiled, so not really sure when it would get called, but it seems to specifically check for "num_world == 6 && num_level == 3". I'm not sure what they did with all of these level checks and such in the EDU games.

It sounds strange that it's called from the init code all right, but I just realized that there's a Photographer in the middle of the reversed section in the original game, so it must be there to handle respawning at that particular location. Anyway, the num_level check must have been removed from EDU and Designer, since it appears in level 54 in the former, and can appear in any level in the latter. Maybe the num_world check was removed as well, and the call from INIT_RAY got confused by the Lava somehow.

Regarding the UFO, makes sense! It's indeed weird though about them keeping stuff like that in Designer. I also just realized now that some of the .wld files in EDU go above 1.3 GB! I remember a few months back Ryemanni and I tried moving a bunch of DES/ETA between .wld files in R1 and noticed there was a limit where it couldn't be bigger than around 900mb. It seems they increased this greatly in later games. Would be interesting to see how much you can actually fit into a .wld in Designer

900 kB, right? I wonder if that's related to the availability of the "X mode" I remember hearing about before – I seem to recall this would be activated on older processors that couldn't handle the full game? In that case, the limit may come from the need to support older processors…

[RayCarrot]

Ah right, I forgot the spell stays when you die in R1. That makes sense indeed!

Around 900kb, yeah. But we never found the exact size. I just remember that including Dark Rayman's DES in the Jungle .wld didn't work, which is why our Dark Rayman mod had to use Rayman's DES That mode you're talking about is probably the one which makes the tiles lower quality. It was removed in the EDU and Kit games. We also skip that part in the level files more or less (it's a separate tileset).

Btw, I was looking through the R1 code a bit more and realized that TYPE_EDU_LETTRE, TYPE_EDU_CHIFFRE and TYPE_EDU_DIRECTION (which are the last 3 types we got names of from Rayman Advance) actually all appear in the PC version of R1's code too. I guess they were already planning the educational games by then. There is no code to handle them though, they're just there as valid event types. Speaking of, I never understood why TYPE_EDU_DIRECTION is a separate type? It's just a static sign post, right?

[PluMGMK]

Around 900kb, yeah. But we never found the exact size. I just remember that including Dark Rayman's DES in the Jungle .wld didn't work, which is why our Dark Rayman mod had to use Rayman's DES That mode you're talking about is probably the one which makes the tiles lower quality. It was removed in the EDU and Kit games. We also skip that part in the level files more or less (it's a separate tileset).

Ah yes, I remember that now. Does anyone know what causes that mode to activate? It feels like something that's there for 286/386 backwards compatibility, but I'm not sure why…

EDIT: Actually, I see the game used PMODE/W, which seems to abstract away all the memory-management concerns I was thinking of, and require at least a 386 to run. Interesting.

Btw, I was looking through the R1 code a bit more and realized that TYPE_EDU_LETTRE, TYPE_EDU_CHIFFRE and TYPE_EDU_DIRECTION (which are the last 3 types we got names of from Rayman Advance) actually all appear in the PC version of R1's code too. I guess they were already planning the educational games by then. There is no code to handle them though, they're just there as valid event types. Speaking of, I never understood why TYPE_EDU_DIRECTION is a separate type? It's just a static sign post, right?

Yep. In fact, it seems they occasionally just used TYPE_PANCARTE instead. Maybe they were originally planning for it to do something special, like the way TYPE_EDU_LETTRE can be collected when a MOT is active. I guess it makes sense that they were already planning the educational games. Do you think that's also why there are red antitoon sprites available in Jungle (albeit not as clean as the EDU ones)?

[RayCarrot]

Ah yes, I remember that now. Does anyone know what causes that mode to activate? It feels like something that's there for 286/386 backwards compatibility, but I'm not sure why…

I haven't actually activated the mode myself, but Rib said he got it to activate by lowering the amount of vram the game had access to in DOSBox.

Yep. In fact, it seems they occasionally just used TYPE_PANCARTE instead. Maybe they were originally planning for it to do something special, like the way TYPE_EDU_LETTRE can be collected when a MOT is active. I guess it makes sense that they were already planning the educational games. Do you think that's also why there are red antitoon sprites available in Jungle (albeit not as clean as the EDU ones)?

Interesting! Regarding the red antitoons, probably. I think the reason they appear so broken is due to the incorrect palette being used. If I remember correctly in the EDU games antitoons with HP set to 2 are invincible and insta-kill you. I tried doing that in R1 once and it sadly didn't work (they just took two hits to defeat). So while the sprites are there I guess the functionality isn't.

[PluMGMK]

Yep, they use the HP value to determine whether the antitoon is insta-kill or not. I once tried the same thing in R1 and somehow came away with the impression that it was ignored completely. I just tried again with HP=5 though and, yes, it took 5 hits to defeat. Tbh it's actually pretty cool to discover that the original antitoon code actually used the HP field in the way it was intended to be used!

Thinking about the WLD file size situation, it would indeed be cool to try to find the maximum size for a WLD in EDU/Designer. I might just start by trying to port everything over at once and see what happens. I just realized that this arbitrary cutting is similar to what happened to the soundtrack on Rayman Forever. Ostensibly it was done to save space, but when you look at the hard facts, that doesn't actually turn out to be a legitimate reason (if I interpret Rib's comment here correctly).

EDIT: Well, I can confirm that 1.5 MB is too big anyway! It doesn't just crash the game, it actually crashes DOSBox altogether!

EDIT2: OK, either I did something stupid in the porting code, or the limit is much lower on Designer than on EDU. Even an 847-kB WLD crashes it.

EDIT3: Nah, it's definitely a limit, not my doing something stupid. If I port over just the UFO stuff to Cave (from EDU), it works fine. So I guess EDU has much bigger WLD-size limits than Designer. I wonder if Skops would fit in EDU…

[RayCarrot]

Oh wow, I just now noticed my typo where I wrote the .wld size in GB

Too bad there's still a limit for it in Designer I was hoping it'd be much higher. I wonder if it'd be possible to increase somehow.

[PluMGMK]

Not sure what type of limit it is though. It seems to read in the first 500~700 kB of the file (depending on the order in which I port over the ETAs and DESs) before seeking backwards by some multiple of 512, and then subtracting some other multiple of 512 from the current position and seeking to there. It then reads some other number of bytes and gives up the ghost. I guess the reason is that it realizes the next ETA/DES beyond that initial 500~700 kB will take it over whatever limit is coded, but it still seems like strange behaviour to me.

EDIT: What's even stranger is that the limit in Designer is clearly even less than that in Rayman 1. Anything over 800 kB seems to crash Designer (based on this I'm guessing the exact limit is 768 kiB), whereas the Rayman 1 files seem to range from 772 kB (Mountain) to 995 kB (Cave). Why would they impose such strict limitations on themselves?

[RayCarrot]

That is odd behavior indeed. I was looking through the code and found this:

Code: Select all

TailleMainMemTmp = 139264;
TailleMainMemFix = 317440;
TailleMainMemWorld = 1536000;
TailleMainMemLevel = 1536000;
TailleMainMemSprite = 914432;

It runs when the game launches and it sets up a lot of variables and environment stuff. I think these might be the max sizes for each memory block. They appear higher than what is allowed in the files though, but there might be other data included here (like "world" in this case might include the sounds for example), but then I'm not sure why "Sprite" is separate. Maybe that's what's causing the limit? If it is this simple I believe a simple hex edit should be enough.

Then when the game actually starts loading data it seems to allocate the memory with the sizes specified above (init_memory is like a wrapper around malloc the game uses for some things):

Code: Select all

init_memory((int)&main_mem_sprite, TailleMainMemSprite);
init_memory((int)&main_mem_level, TailleMainMemLevel);
init_memory((int)&main_mem_world, TailleMainMemWorld);
init_memory((int)&main_mem_fix, TailleMainMemFix);

So the seeking might be some weird side effect of what happens when it's out of bounds of the allocated memory? Not sure as I'm not very familiar with those things.

Edit: Sadly I can't find anything like this in the Designer exe :/ I'll try this with the normal R1 exe later.

Edit 2: Ah, it also seems the PC version of R1 does it slightly differently. I actually got the above code from Rayman Classic. Here it is on PC:

Code: Select all

  if ( availableMemory < 0x200000 )
    sub_1F988((int)aNotEnoughMemor);
  if ( availableMemory >= 0x300000 )
  {
    byte_CFA33 = 1;
    byte_CFA32 = 1;
    byte_CFA17 = 1;
    byte_CFA13 = 0;
    byte_CFA18 = 1;
    byte_CFA1A = 1;
    TailleMainMemTmp = 139264;
    TailleMainMemWorld = 1002496;
    TailleMainMemLevel = 556032;
    TailleMainMemSprite = 914432;
    TailleMainMemFix = 317440;
  }
  else
  {
    byte_CFA33 = 0;
    byte_CFA18 = 0;
    byte_CFA1A = 2;
    TailleMainMemTmp = 95232;
    TailleMainMemWorld = 404480;
    byte_CFA32 = 1;
    byte_CFA17 = 1;
    byte_CFA13 = 1;
    byte_CFA20 = 1;
    TailleMainMemFix = 317440;
    TailleMainMemSprite = 1228800;
    TailleMainMemLevel = 385024;
  }

This also seems to be what determines if the game runs in the low graphics mode that we discussed before.

Edit 3: OMG, it worked! I did a hex edit based on the PC values in the R1 exe file and the game loaded with my 1101 kb .wld file which previously crashed!

[PluMGMK]

Fantastic! Sounds like a command-line patcher (or something similar) could be made for this! Incidentally, what's the best way to disassemble the R1 PC exe?

On a side note, do you have any idea why they didn't put things like Antitoons and Hunters in ALLFIX? Seems like a waste of disk space to have them duplicated in all the WLDs.

[RayCarrot]

Perhaps yeah. Problem with R1 is the exe is compressed (same for EDU, Kit etc.), so it needs to be decompressed first. It's a bit of an obscure program needed for it, so I can send it to you if you'd like. As for decompiling, sadly Ghidra doesn't support the exe. Only one I know which does is IDA.

Regarding the Hunter and Antitoons, no idea. I've wondered myself why it's organized the way it is and I guess it probably has to do with them originally being more specific to certain worlds. Interestingly the antitoons are actually in allfix in the Jaguar version.
While on the topic of this, I've always found it interesting about the platforms. To me it seems the developers originally intended for every world to have its own platform graphic, but then for some reason ended up using clouds everywhere. For example Jungle has the water lily, Music has clouds (since it was originally sky themed), Mountain/Cave have these spiky platforms (and the unused rounded ones), Image has the pencil sharpeners and Cake has the candy platforms (at least on Jaguar and in Designer). Granted these platforms are only of the normal stationary type, so clouds could be used in more ways. But yet they never used the clouds in the Jungle world...

Edit: I just had another look in the Designer exe and while I found where it sets the same max memory values it seems it's a lot more complicated this time around. They are no longer just constant values, instead it gets them from somewhere - not sure where yet :/.

Edit 2: Unrelated, but you might find it interesting: the RayKit exe actually has a ton of unreferenced functions leftover from when Ubisoft developed Rayman 1. There are functions to compile all sorts of files, like the .wld files etc. as well as some graphics editors or converters. Not sure how to enable any of it, but it's cool it's all still there! Also, in the list of power-ups it mentions Super-Seed, which is unused and crashes the game if used.

[PluMGMK]

Perhaps yeah. Problem with R1 is the exe is compressed (same for EDU, Kit etc.), so it needs to be decompressed first. It's a bit of an obscure program needed for it, so I can send it to you if you'd like. As for decompiling, sadly Ghidra doesn't support the exe. Only one I know which does is IDA.

Since I'm off work, just for fun I decided to roll my own decompressor. I took the DECODE.ASM from the source archive available at the PMODE/W site I posted before, and wrapped it in some code to make it a standalone Linux executable. It sorta works, in that IDA can make sense of the output, but since I just dumped everything in a file without any relocation info, I'm not sure how useful it is to me or anyone else. Also, it contains strings like "MOU;TAI;", which makes it feel like there were mistakes in the decompression. Anyway, please do send me the tool if you can, at least so I can see if my attempt worked properly.

Regarding the Hunter and Antitoons, no idea. I've wondered myself why it's organized the way it is and I guess it probably has to do with them originally being more specific to certain worlds. Interestingly the antitoons are actually in allfix in the Jaguar version.

That's interesting indeed. I can see why they would have originally had Hunters in mind for one or two worlds, but Antitoons… I guess it's yet another weird decision they made with the multi-platform engine and ports!

While on the topic of this, I've always found it interesting about the platforms. To me it seems the developers originally intended for every world to have its own platform graphic, but then for some reason ended up using clouds everywhere. For example Jungle has the water lily, Music has clouds (since it was originally sky themed), Mountain/Cave have these spiky platforms (and the unused rounded ones), Image has the pencil sharpeners and Cake has the candy platforms (at least on Jaguar and in Designer). Granted these platforms are only of the normal stationary type, so clouds could be used in more ways. But yet they never used the clouds in the Jungle world...

That struck me as well, especially after playing Redemption. Ryemanni seems to have followed through with their original intention (though I can't remember how many worlds clouds were actually in…). I guess they felt clouds were suitable to both Music and Mountain, since they're both high in the sky, and then got so used to using them that they just started putting them in all the worlds. I guess Jungle, being the first world, was more established at the time that this happened, so everyone knew to keep using waterlilies there. Not sure how this theory stacks up if the natural worlds were done before the abstract ones (including Music) though…

Edit: I just had another look in the Designer exe and while I found where it sets the same max memory values it seems it's a lot more complicated this time around. They are no longer just constant values, instead it gets them from somewhere - not sure where yet :/.

Agh! So near, and yet so far…

Edit 2: Unrelated, but you might find it interesting: the RayKit exe actually has a ton of unreferenced functions leftover from when Ubisoft developed Rayman 1. There are functions to compile all sorts of files, like the .wld files etc. as well as some graphics editors or converters. Not sure how to enable any of it, but it's cool it's all still there! Also, in the list of power-ups it mentions Super-Seed, which is unused and crashes the game if used.

Wow! Makes me wonder how big the "Kit" actually was when they used it internally. If all that stuff was built into the game, I guess they probably didn't make much, if any, use of Windows-based tools. I wonder if they used the Windows-based Mapper (that got released) or if they had another built-in tool that was more usable than Tiles Mode in the Events Editor…

[RayCarrot]

I sent the program

Regarding Designer, I did find some strings which suggested there was a tile editor in the game editor. Also something called "init editor". No clue what that is. It's from these two strings:
"EVENT editor * MAP editor * INIT editor"
"Enter/Quit the init editor"
Could be related to building the game files or something, not sure. But yeah, it seems likely they used this Kit tool for pretty much everything.

[PluMGMK]

Thanks for that! My own decompressor clearly wasn't cutting the mustard because now I see way more strings.

I'm still looking around, but I noticed the EDU EXE contains the strings "BONNREP", "GENDOOR" and "CHERCHE", which must mean that they were hardcoding Gendoor sounds at one point, like they did in the released version of Designer. Probably a temporary measure, before they got the VoiceLine code working…

[PluMGMK]

I have some news – I managed to implement memory loading on Linux, in case anyone's interested! Would you be interested in taking that upstream? The "bad" news is that I had to compile a 32-bit version of Dosbox, whereas I think the trend is towards 64-bit-only distros (cf. Ubuntu). I'm not sure if adding 64-bit pointer support to Ray1Map is worth the effort – I suppose it depends on whether or not 32-bit versions of Dosbox and/or other emulators continue to be available in the future.

Finding the base pointer at run-time is easy on Linux if you have the debugging symbols available btw. You can attach gdb to the running Dosbox process and find the address of MemBase and convert it to decimal – something like this:

Code: Select all

(gdb) x &MemBase
0x98cd660 <MemBase>:	0x00000000
(gdb) print 0x98cd660
$1 = 160224864
(gdb) quit

Of course, when Dosbox is actually running, the 0x00000000 will be an actual memory location instead, but either way it's the leftmost number that matters.

The same technique could be applied to any emulator (e.g. PCSX), but I haven't yet figured out if it could be implemented within Ray1Map itself, or if it's possible on Windows.


On the (de)compression side, I've actually still been holding out hope for a freely-distributable command-line patcher. I've written a tool in Rust that can analyse and decompress a PMW1 executable, using the freely-available code and documentation of PMODE/W. I've also been working on figuring out what actually needs to be patched in the RayKit EXE, but I still haven't found it yet. The biggest problem is that, once it's patched, it needs to be either converted to LE format, or recompressed, in order to actually run. I've tried to write a recompressor but it keeps making mistakes that render the output useless, probably due to an off-by-one I can't find.

[RayCarrot]

Awesome! Forgive my lack of Linux knowledge, but is this something you have implemented into Ray1Map directly now so it automatically works? If so, I'd love for it to be a part of the repo if you're fine with it - if so you can either make a pull request or I can see if I can get in touch with Ad to add you as a collaborator.

Regarding other emulators, I would absolutely love that, and I know it's possible since I tested it with some hard-coded pointers. Problem is none of the emulators I know of have static pointers that I could find. BizHawk, which would be the most ideal to support since it has different cores for different consoles (although this might not work on Linux since I think it uses .NET for the UI), has an API I looked into. And while it does support reading game memory through it, it'll probably be too slow since it's not really built for reading every frame... Ideally we'd want to read from memory directly. But if we can find a way to get static pointers then it'll be incredibly easy to implement in Ray1Map All R1 versions look practically the same in memory, so the existing code will work.

Regarding compression, I don't think you need to re-compress the exe. When I did my edit in R1 I used the decompressed exe and it loaded fine.
Regarding RayKit, if you're analyzing the exe I would recommend you also check the Rayman Classic exe. It has function names and such, so it's really helpful for cross-referencing certain things. I was even at one point going making an IDA database for the PC versions where I was copying over the function names from Classic, but I never got very far as it was very time consuming

[PluMGMK]

I've added Linux code to ProcessMemoryStream.cs, using the ptrace syscall, but for now the pointer still needs to be found manually.
I just realized you don't actually need debug symbols though – the string "MemBase" is embedded in the executable itself, so it probably is possible to look it up at runtime automatically within Ray1Map, at least on Linux! Still not sure about Windows though. I'll play around with it and submit a PR once I'm happy with it. I can try to poke around other emulators too if I get it working.

The thing about my decompressor is that it currently produces an uncompressed EXE in the PMW1 format, whereas PMODE/W expects PMW1-format executables to be compressed (unless it's compiled with a special switch). The other decompressor you sent me converts the EXE to LE format after decompressing, which I don't think is 100% straightforward, so I opted for recompression. In hindsight though, converting to LE format would probably be the easier course to take. I might take another look at doing that!

I guess the Rayman Classic Android exe is in ELF format, which is why it has all those names, so it's the very same reason that the MemBase pointer is so easy to find at runtime in Dosbox on Linux. Doesn't bode well for Windows then, but I'll play around with GDB under Wine and see if anything turns up. Would you mind sending me the Classic exe? I know it's freely available, but I'm running LineageOS on my phone with no Google apps so I'd have to jump through hoops to get it the "normal" way.