Rayman 1 Level Editor/Viewer (Ray1Map)

Discuss tools to aid in the modification and running of Rayman games.

Moderators: English moderators, Modding and utilities team

PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

Ah yes, despite the way it looks generalized, the game does of course read it into a fixed-size buffer and use fixed pointers to address it. That's annoying…

As for Bzzit/Moskito's hitzone, well, I can't find out. When I put either Bzzit or Moskito in a Designer level, the game slows to a crawl, and "illegal read"s keep popping up in the Dosbox debugger, coming from the "display2" function. There's some pointer that keeps being incremented, and moves off the end of the address space. I'm too tired to investigate right now, but that's damn annoying – I've wanted to have Moskito in Designer for over a decade…

EDIT: For what it's worth, this is what I was getting in the Dosbox debugger, and the offending code displayed in IDA Freeware (stack variable "var_3C" contains the offending pointer):
snip-1601237680.png
That "FatalError" section isn't really relevant, but it's how I got my bearings that this was the "display2" function…
That "FatalError" section isn't really relevant, but it's how I got my bearings that this was the "display2" function…
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

I was thinking more about Mr Skops hitbox. I was curious if it's show the big vertical one preventing you from jumping over him.

Regarding the errors for Bzzit/Moskito, that's odd. There is one animation for them using some unused sprites IIRC, maybe it's trying to play that and is thus getting wrong addresses? But then again I don't think any state points to it, so that'd be weird.

Edit: Btw, I just did some changes to the memory loading to support BizHawk. Since it uses a 64-bit base address I had to change the system a bit to have a base address everything is relative to (to avoid completely reworking the Pointer class to support 64-bit pointers). Could you please check to make sure it still works on Linux when you got a chance?

Edit 2: I've now reimplemented saving support for PC :)
Image
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

RayCarrot wrote: Mon Sep 28, 2020 9:36 am I was thinking more about Mr Skops hitbox. I was curious if it's show the big vertical one preventing you from jumping over him.
Ah, that! No, it's actually just his feet:
snip-1601320766.png
RayCarrot wrote: Mon Sep 28, 2020 9:36 am Regarding the errors for Bzzit/Moskito, that's odd. There is one animation for them using some unused sprites IIRC, maybe it's trying to play that and is thus getting wrong addresses? But then again I don't think any state points to it, so that'd be weird.
So I tried again, this time copying the ETA and DES from EDU instead of R1, and, guess what, it worked fine! So there's actually a difference between the R1 and EDU ETA/DES for Moskito, to account for some change in the engine. I wonder if that means that they were planning on including him in EDU at some point! :o He's the only boss whose sprites they didn't delete, after all…

EDIT: Nevermind, I think I just ported over the wrong ETA. It looks like my code specified the wrong number of allfix ETAs for R1. That might explain why Skops' hitbox isn't as expected too. :oops2:

EDIT2: Oh man, there were two mistakes in my code around the ETAs, which cancelled each other out to make porting from EDU work fine, but not from R1! :fou2:

EDIT3: OK, so now the correct MST.ETA copied from R1 works half the time (i.e. I can create and fight a Moskito boss), but other times it seems to send the instruction pointer itself off the end of the address space. No clue how that could be happening! :fou2: The corrected SKO.ETA now produces a sleeping Skops with no hitbox at all. I'll have to see what happens when I add his commands in!
RayCarrot wrote: Mon Sep 28, 2020 9:36 am Edit: Btw, I just did some changes to the memory loading to support BizHawk. Since it uses a 64-bit base address I had to change the system a bit to have a base address everything is relative to (to avoid completely reworking the Pointer class to support 64-bit pointers). Could you please check to make sure it still works on Linux when you got a chance?

Edit 2: I've now reimplemented saving support for PC :)
Right now, I just get GUIErrors when I try to use memory loading. I hope something didn't get screwed up when I merged it into my tree… :?
But anyway, 64-bit pointer support sounds promising, and it was a prospect which daunted me back when I first went at memory loading. Hopefully this means 64-bit Dosbox builds will work too! :)

I'm actually considering rewriting the Linux memory loading anyway, to use process_vm_readv and process_vm_writev instead of ptrace. The ptrace-based implementation felt a bit sluggish when I tried it again the other night, possibly because my current Dosbox build includes the debugger (which can't be turned off at runtime). I just need to figure out how to pass structs to syscalls from C#, which shouldn't be too hard…
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

Ah okay, glad you figured it out. It's easy to make mistakes with the indexes. it gets a bit confusing when you have to take into account the ones in allfix as well. That most likely explains Skop's incorrect hitbox too. There is one byte in the ETA which determines some hitbox related things.

Regarding the memory loading, weird that you're getting errors... You have the latest commit merged, right? I tried it again and it works fine for me, so maybe something with the Linux specific code in which case I sadly can't really test it.
Image
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

When I select "Load from memory", I'm getting
GUI Error: You are pushing more GUIClips than you are popping. Make sure they are balanced.
UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr) (at /home/bokken/buildslave/unity/build/Modules/IMGUI/GUIUtility.cs:197)
And nothing beyond the "Default memory options" dropdown appears in the settings window. Maybe it's related to the extra buttons I added for the DES/ETA porting. I'll try a clean copy of your branch and see what happens…

EDIT: That doesn't seem to have helped either. I'll have to dig into it…

EDIT2: I got it to work by commenting out this if statement:

Code: Select all

if (DefaultMemoryOptionsIndex != PreviousDefaultMemoryOptionsIndex) 
            {
                if (PreviousDefaultMemoryOptionsIndex == -1) {
                    var match = Enumerable.Range(0, DefaultMemoryOptionDisplayNames.Length).FirstOrDefault(x => 
                        DefaultMemoryConfigs[x].IsPointer == Settings.IsGameBaseAPointer &&
                        DefaultMemoryConfigs[x].ProcessName == Settings.ProcessName &&
                        DefaultMemoryConfigs[x].ModuleName == Settings.ModuleName &&
                        DefaultMemoryConfigs[x].Offset == Settings.GameBasePointer);

                    DefaultMemoryOptionsIndex = match + 1;
                }
                
                PreviousDefaultMemoryOptionsIndex = DefaultMemoryOptionsIndex;

                if (DefaultMemoryOptionsIndex != 0) 
                {
                    Settings.IsGameBaseAPointer = DefaultMemoryConfigs[DefaultMemoryOptionsIndex - 1].IsPointer;
                    Settings.ProcessName = DefaultMemoryConfigs[DefaultMemoryOptionsIndex - 1].ProcessName;
                    Settings.ModuleName = DefaultMemoryConfigs[DefaultMemoryOptionsIndex - 1].ModuleName;
                    Settings.GameBasePointer = DefaultMemoryConfigs[DefaultMemoryOptionsIndex - 1].Offset;
                }
            }
Now I just need to figure out why I needed to do that… :P

Anyway, memory loading still works for 32-bit Dosbox on Linux. :) Not 64-bit though, I guess that'll take more work on my part!
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

Hm, that's odd. Try putting a try/catch around it and log the exception. It's annoying Unity doesn't actually show it correctly...

Regarding 64-bit DOSBox, the groundwork is there but it needs to be implemented yeah. If it's just that the address it reads is 64-bit we could make a toggle in the settings where you decide if you should read an int or a long for the base memory address.

Edit: Btw, something interesting I just found you might be interested in. So I was looking into the files for custom Designer levels (ray1map doesn't fully support them yet, it just reads the temporary .cmd files the Designer editor autogenerates) and turns out it's the Jaguar format :O Events.mev is the "MapEvent" struct from Jaguar, which maps all events to a grid in the level (and it's oddly big endian just like on Jaguar huh) and Events.sev is the "OffList" struct as well as the actual list of event instances (known just as "Event" in Jaguar). This file is in pure text form though.
Image
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

OK, I uncommented the block, and the issue has magically disappeared! I think it must have been to do with stale settings stored from an older version, which were sending that code down unforeseen paths… :?

I'll definitely look again at getting 64-bit Dosbox working! :D
EDIT: Got it! :D I've made it automatically determine whether the attached process is 32-bit or 64-bit, and decide whether or not to read a long. It works 100% on Linux, and I've written some Windows code that should work too, but please check it.
EDIT2: But as for switching from ptrace to process_vm_readv, I don't think I can do that without using "unsafe" code, since it asks the kernel to write data directly into my process's address space. I might experiment with that on a separate branch, to see if it actually improves performance. I obviously don't want to bring unsafe code in lightly!

That's funny about the Jaguar format! I was just thinking the other day, as I was using the events editor, that it's a bit like the Jaguar engine in that events can be added and removed at runtime – I wonder if that's related to this. It's mad that it's big-endian though, I wonder what sort of code they wrote for that!
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

I just saw your pull request. There were a few syntax issues with the way the out param was used for the IsWow64Process function which I fixed. I also flipped the win64 boolean after calling IsWow64Process on the current process if Unity is not 64-bit since the value IsWow64Process returns is if it's on Wow64 and not if it's 64-bit. I added some logs and made some tests and it seems to work. :) I however don't have a 64-bit version of DOSBox to test it on. Where can I find that?

Here's how I have it looking now:

Code: Select all

#if ISWINDOWS
#if UNITY_64
	    const bool win64 = true;
#else
            IsWow64Process(Process.GetCurrentProcess().Handle, out var win64);
            win64 = !win64;
#endif
            IsWow64Process(processHandle, out var isWow64);
            is64bit = win64 && !isWow64;
            UnityEngine.Debug.Log($"Is current process 64-bit: {win64}");
            UnityEngine.Debug.Log($"Is game process 64-bit: {!isWow64}");
Image
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

There's a 64-bit DOSBox build included in my TPLS build: http://www.vigovproductions.net/patched-dosbox-v4.zip – its MemBase pointer is 51652992 (that took me way longer to figure out than it should have :oops2:).

The idea with the win64 variable is to find whether the host OS is 64-bit. If UNITY_64 is defined then it means Unity itself is 64-bit, so the OS is obviously 64-bit. Otherwise, if UNITY_64 is not defined, Unity is 32-bit, which is why I'm calling IsWow64 on the current process handle. If IsWow64 is true for the 32-bit Unity process, that means the host OS is 64-bit, otherwise it's 32-bit. It's a very confusing API function, but I think that's the logic I captured by running it my way…
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

Thanks! The 64-bit version worked fine :) I should mention the sound was crackling when changing levels... I remember someone else said the patched TPLS DOSBox did this. Is this anything you've noticed too?

Regarding the 64-bit value, I see. It's indeed a bit confusing :P I'll remove the flipping and merge the branch with my out param fix then.

Edit: I also got the settings window exception you got and fixed it now :)
Image
RaymanRedemption
Obus à Pattes
Posts: 77
Joined: Tue Aug 25, 2020 3:55 pm
Tings: 385

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RaymanRedemption »

It looks amazing work!
I'm asking if there is a link to download and install it?

(I already have the officiel Rayman Designer game on my computer)
Hâte de jouer au prochain jeu Rayman ReDesigner !
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

Not at this time no. We haven't made any builds since most features like the settings, tools etc. are all in the Unity editor. You can still check out and use Ray1Map if you download the source code and open it in Unity, but it's not very user friendly atm.
Image
RaymanRedemption
Obus à Pattes
Posts: 77
Joined: Tue Aug 25, 2020 3:55 pm
Tings: 385

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RaymanRedemption »

RayCarrot wrote: Fri Oct 02, 2020 10:39 am Not at this time no. We haven't made any builds since most features like the settings, tools etc. are all in the Unity editor.
Thanks for answer!
Do you plan to build it or it is an experimental tool?

Maybe this project is used for Rayman ReDesigner?
Hâte de jouer au prochain jeu Rayman ReDesigner !
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

Soon you'll be able to view maps and such on the web, but as for an actual build with editing support I don't know. It all depends on if we find someone who's willing to help out with the UI. Right now this is more of a technical tool than an actual editor sadly enough.
Regarding ReDesigner, no, that's completely separate.
Image
Hunchman801
Bad Rayman
Posts: 79055
Joined: Thu Aug 07, 2003 6:50 pm
Location: Paris, France
Contact:
Tings: 569889

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by Hunchman801 »

RaymanRedemption wrote: Fri Oct 02, 2020 12:10 pm Maybe this project is used for Rayman ReDesigner?
Rayman Redemption was built with Game Maker and therefore uses a completely different engine than the original game (the one that Ray1Map supports), and it will most likely be the same for ReDesigner.
insert sig for RPC's 17th bday
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

So I've finally got the porting code into a state for which I'm happy to submit a PR. :)

That thing I mentioned earlier about Moskito started happening consistently btw. Despite the fact that I had the correct ETA, the instruction pointer would somehow jump off the end of the code segment:
Image

I then realized that it might be due to using compiled rather than local commands in my MLT, and changing them to (what I think are) local equivalents did indeed seem to help. Here's what I have now:

Code: Select all

œdef,MS_mst2,MST2, 4 ,
MST.ETA,

 11 ,     1 ,
 19 ,    1,
 12,     1,
 11 ,     2 ,
 30 ,    200,
 12,     7,
 33 , 255,
0,0,    /bidon/
 0 , 13 ,
80,80,48,
0,0,12,227,254, 5 ,
As opposed to:

Code: Select all

œdef,MS_mst2,MST2, 4 ,
MST.ETA,

 11 ,     1 ,
 19 ,    1,
 23,     0,
 11 ,     2 ,
 30 ,    200,
 24,     1,
 33 , 255,
0,0,    /bidon/
 0 , 13 ,
80,80,48,
0,0,12,227,254, 5 ,
Adding objects for Kit doesn't seem to work in Ray1Map at the moment because DecompileCommands isn't implemented, which is what made me think of this. Is decompiling commands difficult or is it just something that nobody has gotten around to?
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

Nice work! I went ahead and merged it now :) I also tried optimizing it slightly by not recreating the DES and ETA arrays for each time you add one, but rather use a temporary list to add to first.

Regarding the DES/ETA names, I think it might be better if we merge the R1/EDU columns, or was there a reason to separate them? This was it'd be more of a general way of matching a DES/ETA entry with its index, which means we could expand it for PS1 and such later. We could probably also just go with a single entry in each cell, rather than one per world (since we can check world availability in the normal DES/ETA index columns).

Regarding the commands, I haven't gotten around to adding support for decompiling yet. Tbh the commands are a bit confusing to me with all of these offsets :oops2: But what you did fix the commands in your example looks good.

It would be nice if we could also update the .mlt files automatically now, with info from the sheet we use. There is already a system for reading it, and implementing writing there shouldn't be too hard. We'd of course loose the developers comments and such since padding is ignored, but that shouldn't matter. One problem might be the memory though, as like I mentioned it seems they hard-coded the memory size for allocating that file. But we might be able to edit the exe to fix that.
Image
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

RayCarrot wrote: Mon Oct 05, 2020 7:59 pm Regarding the DES/ETA names, I think it might be better if we merge the R1/EDU columns, or was there a reason to separate them?
Umm… :oops2: Yeah, that makes sense. I probably separated them based on the notion I mentioned before that the files were different between R1 and EDU, even though I soon figured out it was my mistake. :oops2:
RayCarrot wrote: Mon Oct 05, 2020 7:59 pmThis was it'd be more of a general way of matching a DES/ETA entry with its index, which means we could expand it for PS1 and such later. We could probably also just go with a single entry in each cell, rather than one per world (since we can check world availability in the normal DES/ETA index columns).
Makes sense, though we'd have to separate a few things, like the Image versus Cave levers (LEV2 versus LEV). That should probably be done anyway since they're so different!
RayCarrot wrote: Mon Oct 05, 2020 7:59 pm Regarding the commands, I haven't gotten around to adding support for decompiling yet. Tbh the commands are a bit confusing to me with all of these offsets :oops2: But what you did fix the commands in your example looks good.
Fair enough!
RayCarrot wrote: Mon Oct 05, 2020 7:59 pm It would be nice if we could also update the .mlt files automatically now, with info from the sheet we use. There is already a system for reading it, and implementing writing there shouldn't be too hard. We'd of course loose the developers comments and such since padding is ignored, but that shouldn't matter. One problem might be the memory though, as like I mentioned it seems they hard-coded the memory size for allocating that file. But we might be able to edit the exe to fix that.
Sounds good, although that would also involve decompiling commands, right? I don't think the comments really matter, since we've probably learned all we can from them at this point! :P
Right now, I could write another patcher in Rust to edit the EXE, but ideally we'd do it from within Ray1Map I guess. That would involve rewriting the compression and decompression algorithms in C#. The algorithm code is quite difficult to work with, especially the decompression bit, which I just copied from the original ASM straight into an inline block in Rust! :P That won't work here though! :fou:

I thought of something else – one of the always events in Mountain has "TYPE_ETINC" as its object_type in the MLT, and it works. Do you know where the game gets the information it needs to turn this into a number? The string "TYPE_ETINC" doesn't seem to be in the EXE…
RayCarrot
Uglette
Posts: 2134
Joined: Sat Jan 11, 2014 5:46 pm
Contact:
Tings: 29129

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by RayCarrot »

PluMGMK wrote: Mon Oct 05, 2020 9:49 pm Makes sense, though we'd have to separate a few things, like the Image versus Cave levers (LEV2 versus LEV). That should probably be done anyway since they're so different!
Yeah, we'd have to separate any events which use different DES or ETA between worlds, but I don't think there are so many of those luckily.

PluMGMK wrote: Mon Oct 05, 2020 9:49 pm Sounds good, although that would also involve decompiling commands, right? I don't think the comments really matter, since we've probably learned all we can from them at this point!
Right now, I could write another patcher in Rust to edit the EXE, but ideally we'd do it from within Ray1Map I guess. That would involve rewriting the compression and decompression algorithms in C#. The algorithm code is quite difficult to work with, especially the decompression bit, which I just copied from the original ASM straight into an inline block in Rust! That won't work here though!
The commands would have to be decompiled yeah, since we use the compiled variants in the sheet. it might be cleaner using the decompiled versions since we don't need the label offsets then, but I thought since you're mostly editing compiled levels with ray1map it would be faster to have them in the compiled format directly.
Regarding patching the exe, the problem might be that not all exe files are the same, so the offset to edit would differ. For example there's the Rayman Gold version and then some 1-language versions.
PluMGMK wrote: Mon Oct 05, 2020 9:49 pm I thought of something else – one of the always events in Mountain has "TYPE_ETINC" as its object_type in the MLT, and it works. Do you know where the game gets the information it needs to turn this into a number? The string "TYPE_ETINC" doesn't seem to be in the EXE…
Any event definition in the .mlt files which use names are unused. You can see that TYPE_ETINC is used with MS_BB1 if you check the if definition, which is Mr Stone. If you check the Jungle .mlt you can also see they left in a SubEtat name for the fruit Moskito drops named "SE_MST_SHAKY_WAIT", so they originally used names for all states and event types. I wish we had them all... Sadly they removed the actual name tables in the final release. I remember I once tried changing one of the used events to have a name for the type and the game just defaulted it to 0 since it couldn't parse it.
Image
PluMGMK
Ambre Écueil
Posts: 32139
Joined: Fri Jul 31, 2009 9:00 pm
Location: An tír ina labhraítear (beagán ar fad) an teanga ina bhfuil sé seo scríofa
Contact:
Tings: 6075

Re: Rayman 1 Level Editor/Viewer (Ray1Map)

Post by PluMGMK »

Ah! That explains it! I remember in the old days, people added the "Blast" event to their MLTs as non-always, and it seemed to work with TYPE_ETINC, except that it turned into an electoon when it hit a solid type. We assumed that that was a messed-up version of how Mister Stone's blasts turned into clouds, but of course the whole thing was running the Livingstone code (type 0), so of course it didn't work properly! Mystery solved! :fou:
Post Reply