Rayman Pirate-Community

Code: Select all

Rayman says fatal error :
VIGNET.DAT : Can not open (pcx).

Code: Select all

VIGNET.DAT    # VIGNET can stand for Vignettes which obviously means it contains textures

INTRO.DAT     # INTRO stands for Introduction -> intro cutscene

CONCLU.DAT    # CONCLU stands for Conclusion -> outro cutscene

RAYMAN.EXE    # Obivous...

SNDD8B.DAT    # SND stands for sound -> contains sound effects

SNDH8B.DAT    ?? Same guess... but the doesn't contain any sound and doesn't speak much to me when openned with an hex editor

SNDVIG.DAT    # Same guess. This file actually contains voices and misc. sounds I've never heard before

HMIDET.386    # .386 files are often drivers, DET surely stands for Detection. This file is absolutely useless for us.

HMIDRV.386    # Same guess -> DRV stands for Driver. Useless file.

RAY.LNG       ?? LNG maybe stands for Languages, so maybe the file actually contains all of the text of the came but I can't tell

RAYMAN.CFG    ?? CFG obvious stands for Configuration... but what ?

Code: Select all

/*
 * Ray2Get
 * Source Code
 * by Palorifgrodbierzrt 2008
 *
 * Ray2Get is a little command-line tool that allows you
 * to convert Rayman 2 musics (.apm files) to regular
 * .wav files.
 *
 * The .apm files are encoded in ADPCM format. But they
 * can't be converted or even played properly with any
 * standard ADPCM decoder/reader because of their format
 * specification.
 *
 * I used the example code from MultimediaWiki to decode
 * from IMA-ADPCM to 16-bit PCM, and adapted it for a
 * stereo output.
 * I also tried to study the files' headers to get the
 * correct initial values.
 *
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

// The IMA step table
const unsigned short step_table[89] = { 
  7, 8, 9, 10, 11, 12, 13, 14, 16, 17, 
  19, 21, 23, 25, 28, 31, 34, 37, 41, 45, 
  50, 55, 60, 66, 73, 80, 88, 97, 107, 118, 
  130, 143, 157, 173, 190, 209, 230, 253, 279, 307,
  337, 371, 408, 449, 494, 544, 598, 658, 724, 796,
  876, 963, 1060, 1166, 1282, 1411, 1552, 1707, 1878, 2066, 
  2272, 2499, 2749, 3024, 3327, 3660, 4026, 4428, 4871, 5358,
  5894, 6484, 7132, 7845, 8630, 9493, 10442, 11487, 12635, 13899, 
  15289, 16818, 18500, 20350, 22385, 24623, 27086, 29794, 32767 
};

// The IMA index table
const signed char index_table[8] = {
  -1, -1, -1, -1, 2, 4, 6, 8
};

signed int decode(unsigned char nibble, signed short *stepindex, signed int *step, signed int predictor)
{
/*
	The following algorithm has been taken from the Multimedia Wiki,
	and optimized for stereo output.
		-> http://wiki.multimedia.cx/index.php?title=IMA_ADPCM

	Don't ask me how it works...
*/

	signed short diff = 0;

	*stepindex += index_table[(nibble & 7)];
	diff = *step >> 3;
	if (nibble & 1) diff += (*step >> 2);
	if (nibble & 2) diff += (*step >> 1);
	if (nibble & 4) diff += *step;
	if (nibble & 8) diff = -diff;
	predictor += diff;
	if (predictor > 32767) predictor = 32767;
	else if (predictor < -32768) predictor = -32768;
	if (*stepindex > 88) *stepindex = 88;
	else if (*stepindex < 0) *stepindex = 0;
	*step = step_table[(*stepindex)];

	return predictor;
}


int main(int argc, char* argv[])
{
	// Checking the command syntax
	if (argc != 2)
	{
		printf("Error: too");
		if (argc < 2) printf(" few ");
		else printf(" many ");
		printf("arguments.\nUsage: ray2get.exe file.apm file.wav\n");
		exit(EXIT_FAILURE);
	}

	// Variables
	unsigned char nibblel = 0; // The data to decode
	unsigned char nibbler = 0;
	signed int predictorl = 0; // The PCM16 data
	signed int predictorr = 0;
	signed short step_indexl = 0;
	signed short step_indexr = 0;
	signed int stepl = 0;
	signed int stepr = 0;
	unsigned int file_length = 0;
	unsigned int tmp32;
	char pcm_filename[strlen(argv[1]) + 5];
	char* dot_in_filename = NULL;

	// File buffers
	FILE *adp = NULL;
	FILE *pcm = NULL;

	adp = fopen(argv[1], "rb");
	if (adp == NULL)
	{
		printf("Error: unable to open file %s.\n", argv[1]);
		exit(EXIT_FAILURE);
	}

	// Give the apm file's name to the wave file, changing the
	// extension to .wav
	strcpy(pcm_filename, argv[1]);
	dot_in_filename = strrchr(pcm_filename, '.');
	if (dot_in_filename != NULL) *dot_in_filename = 0;
	strcat(pcm_filename, ".wav");

	pcm = fopen(pcm_filename, "wb");
	if (pcm == NULL)
	{
		printf("Error: unable to create output file.\n");
		fclose(adp);
		exit(EXIT_FAILURE);
	}

	// Wave file rendering...
	// RIFF chunk
	fprintf(pcm, "RIFF");
	fseek(adp, 0, SEEK_END);
	file_length = ftell(adp);
	file_length -= 0x62; file_length *= 4; file_length += 36;
	fwrite(&file_length, 4, 1, pcm);
	// Format subchunk
	fprintf(pcm, "WAVEfmt ");
	tmp32 = 16; fwrite(&tmp32, 4, 1, pcm); // Subchunk size
	tmp32 = 1; fwrite(&tmp32, 2, 1, pcm); // PCM
	tmp32 = 2; fwrite(&tmp32, 2, 1, pcm); // Stereo
	tmp32 = 22050; fwrite(&tmp32, 4, 1, pcm); // Frequency
	tmp32 = 88200; fwrite(&tmp32, 4, 1, pcm); // Byte rate
	tmp32 = 4; fwrite(&tmp32, 2, 1, pcm); // Block align
	tmp32 = 16; fwrite(&tmp32, 2, 1, pcm); // Bits per sample
	// Data subchunk
	fprintf(pcm, "data");
	file_length -= 36; fwrite(&file_length, 4, 1, pcm);

	// Reading the initial values
	// (predictor and step index for both channels)
	fseek(adp, 0x2c, SEEK_SET);
	fread(&predictorr, 4, 1, adp);
	fread(&step_indexr, 2, 1, adp);
	fseek(adp, 6, SEEK_CUR);
	fread(&predictorl, 4, 1, adp);
	fread(&step_indexl, 2, 1, adp);

	// Initializing steps for predictors
	stepl = step_table[step_indexl];
	stepr = step_table[step_indexr];

	// Seek for DATA chunk in apm file
	fseek(adp, 0x64, SEEK_SET);

	while (!feof(adp))
	{
		// First nibble decoding, left channel
		fread(&nibblel, 1, 1, adp);
		predictorl = decode(nibblel >> 4, &step_indexl, &stepl, predictorl);
		// Writing output...
		fwrite(&predictorl, 2, 1, pcm); // Writing output...

		// First nibble decoding, right channel
		fread(&nibbler, 1, 1, adp);
		predictorr = decode(nibbler >> 4, &step_indexr, &stepr, predictorr);
		// Writing output...
		fwrite(&predictorr, 2, 1, pcm); // Writing output...

		// Second nibble decoding, left channel
		predictorl = decode(nibblel, &step_indexl, &stepl, predictorl);
		// Writing output...
		fwrite(&predictorl, 2, 1, pcm); // Writing output...

		// Second nibble decoding, right channel
		predictorr = decode(nibbler, &step_indexr, &stepr, predictorr);
		fwrite(&predictorr, 2, 1, pcm); // Writing output...
	}

	fclose(pcm);
	fclose(adp);
	return 0;
}

Code: Select all

Flux 0
Type : Video
Codec : Flic Video (FLIC)
Resolution : 240x150
Framerate : 14

Code: Select all

from	LunaVorax <[email protected]>
to	[email protected]
date	Thu, Apr 1, 2010 at 11:25 PM
subject	CloneKeen development process


Hello !

I don't quite know how to organize that mail but I'll do my best.
As far as I understood, you built the CloneKeen engine from scratch
without any knowledge of the original source code.

I'm from a Rayman community on the internet (https://raymanpc.com)
and I've launched some years ago about trying to rebuild the engine of
Rayman 1 from scratch. However this is something very difficult and I
don't have any knowledge about that.
Time spent and some of the member of the community gained knowledge in
C/C++ (I'm also learning C, I'll start learning C++ soon), but we
still don't know how to use it to work on Rayman 1 engine.

If what I've stated before is true, I'm writing you this mail to ask
you if you could give us some tips and tricks (I don't see another
expression for that) about how to start making our own engine from
scratch.

I'm planning to ask for help to other developers who worked on similar projects.

I thank you very much in advance for your answer,
LunaVorax

Code: Select all

from	Caitlin Shaw <[email protected]>
to	LunaVorax <[email protected]>
date	Fri, Apr 2, 2010 at 2:17 AM
subject	Re: CloneKeen development process




Hi LunaVorax,

This is a very broad topic but I will try to answer as best I can and if you have any further questions feel free to ask.

Yes, I wrote the CloneKeen engine from scratch. I didn't have the source code or use any reverse-engineering techniques such as reconstructing equivalent C functions from the assembly (although that is one way you can do it).

Basically what I did is just to watch very very closely exactly how things behaved, and then write a program that meets those specs. For example, to write the physics of how Keen jumps I would launch the real game, stand next to an object or platform or something, and jump up and down. I'd use the nearby object to measure exactly how high in tiles his head would get before he started back down, and time how long his jump took to complete.

For something like enemy AI, just sit and watch them, stand in front of them and watch them charge you, etc, then write your own AI state for each behavior you observe so that it ends up looking very similar.

You'll find that as you go along questions will pop up where situations could be handled in different ways, and then you can refer back to the original game to see how they handled it. You'll learn a lot of things about the game that you never noticed before this way.

If you're just learning C, I would suggest you might want to start with some simpler games first before working on the Rayman engine. The reason is simple: your first few games are going to be crap, and you don't want to spoil your real project with that junk :). I say this because the code in CloneKeen is not very good by my current standards, and I had been writing C for several years before I started on it.

My usual game I write every time I learn a new programming language is "Wurm" or "Nibbles"-- the top-down game where you are a snake trying to eat randomly-placed apples for points, but each apple you get makes your tail grow longer, and you are not supposed to run into yourself. Writing this game requires real-time keyboard I/O, control of the graphics, a basic game-loop, pretty much everything that is required for any game is present in Wurm, so if you can write Wurm, you have enough grasp of the programming language to write any game in it.

I think Rayman is 2.5D, isn't it? If you haven't ever worked on a 2D game before, I'd suggest making a simple one before you even get started in OpenGL or 3D graphics routines. 2D graphics leave out a number of complications and crazy math you'll have to deal with in 3D, which IMHO is too much to try to take on at once.

Then again, of course, I do not know really how far along you are with C or what kinds of things you have written previously.

You say that rebuilding the engine seems very difficult; here's a good guideline I've used to write MANY programs, not just games, that seemed very difficult at the beginning:

It's kind of like the thing that goes "how do you sculpt an elephant...", and there is another adage that goes like this: "Programming is the art of debugging an empty file.".

Basically, find the biggest most glaring way in which your program does not do what you wish it did. Then work on fixing that. Keep doing it until the dissimilarities are minor enough that you're not too embarrassed to release it.

At first, of course, you might notice that your program just loads and exits, and DOESN'T EVEN OPEN A FREAKIN WINDOW!! What a sucky game! Let's look in the API reference for your desired OS, and figure out how to fix that. Then you might say, hey, that game's window is BLANK!. So you figure out how to draw sprites or whatever in the window, and you add a player character. Great, but your game sucks because you can't move your own player. So you add that in. Then you notice, hey, he's not animated. Then, hey, he's just hovering in mid-air, shouldn't he fall? Then, there's no terrain or map tiles or anything. Then, shouldn't he get blocked by the map tiles and be able to stand on them, etc?

Keep at this, concentrating only on the latest thing that is "wrong" with the program, and before you know it you will have a game. My backups of CloneKeen show that it went from basically a blank screen to a fairly good playable imitation of Keen in about 3 days. The other year or so of development was mostly tweaking and adding features. Once you get that basic engine going, it gets much easier (and more fun, since you can actually see the results of things happening and making progress).

Anyway, that's my Bunch of Talking About Stuff, I hope I helped somewhat; feel free to write back if you have any more, or specific, questions.

Thanks,
Kate

--
Caitlin Shaw <[email protected]>

Code: Select all

from	LunaVorax <[email protected]>
to	Caitlin Shaw <[email protected]>
date	Fri, Apr 2, 2010 at 10:02 PM
subject	Re: CloneKeen development process


Hi Kate,

Thank you so very much for your quick answer, it was complete and funny to read.

I have to say that I did not except to see such a process for the CloneKeen project, this is not reverse engineering but recreation of the engine.

However, if CloneKeen needs the original files in order to work, that means you wrote some code that can open them in order to get the sprites and sound usable (?). This is something I can't figure out. How, then, did you do to open those .CK files ?

I really don't have much question since your mail was extremely clear and concise, it really was a pleasure to read you.

Hope the next answer will be as quick as the first one.
LunaVorax

Code: Select all

from	Caitlin Shaw <[email protected]>
to	LunaVorax <[email protected]>
date	Sat, Apr 3, 2010 at 5:04 AM
subject	Re: CloneKeen development process



Yeah, I did have to open the original files. Actually the first thing I started out with was an app which loaded the first level from episode 1 and displayed it-- as a checkerboard of different colors for the different tile types, because I didn't have the tile graphics loading yet. From there, I added a player sprite, and made him interact with the displayed tiles, and that's how it all started.

It took me some time actually before I got around to figuring out the tile graphics format-- what I did at first was to dump all the graphics out to GIF's using a program which is freely available and used by keen modders for editing the graphics. Then I used Photoshop's "batch convert" feature to convert them all to RAW's and wrote a simple loader which loaded in the image data from the big directory of RAW files, instead of actually opening the file.

Ultimately, the information on the map, graphics, and sound formats I owe to someone who wrote a page documenting pretty much all of it after figuring out how it was done. It was called something like the "Commander Keen for UNIX" page and was an attempt to document the formats in the hope someone would make a rewrite of Commander Keen that could run in UNIX. The page was very old and the one I read was a mirror of the original, which had disappeared previously. I think now the mirror may also be gone as a quick Google search didn't find it.

Anyway, if it were not for that prior work, then given my level of knowledge at the time, I think I probably would not have been into it enough to put forth the effort to figure them out on my own.

Fortunately for me, most of the file formats used in Commander Keen were relatively simplistic. The hardest one was the graphics, which used LZ compression and a weird planar format stemming from the original games use of EGA graphics.

In your case, since you are wanting to recreate a Playstation game, it may be a little tricker, since there is less likely to be modding tools or map/graphics dump tools available already. Also since it is partially 3D, some of the graphics will probably be models instead of sprites, and the map format will likely be more complex.

If you do have to reverse-engineer some or all of the file formats, then you may have to learn PSX assembly and get or build yourself a good Playstation debugger so that you can step through the code as it is loading the data file and try to figure out what means what. You may also be able to figure out some stuff just by looking at the files in a hex editor and observing what values different files, known to be in the same format, place in a given offset.

Most of my reverse-engineering experience has been interested in discovering algorithms, not file formats, so I'm afraid I don't have any more specific tips or tricks here.

If that fails for one or more aspects of the data (graphics, sounds, maps, etc), or, just to get started, you can of course always fall back on what I did with the graphics at first, and just build a library of screencaps or use a modified emulator to dump the models etc currently on screen to a file format that you already understand.

If I were doing it, I would probably start out by learning a little about how PSX games work (in general, since it is likely that each one is a bit different), and take a look at the contents of the CD to see if there was anything there that I thought I could make sense of without too much effort. If not, I would leave the major assembler hacking until later, and start building tools which could efficiently and correctly capture and export the data I needed from a running copy of the game (I would probably modify one of the existing open-source emulators such as ePSXe). I've found I tend to have much more success with things like that when I am doing it to an improve an existing program I can see, rather than wading through hexadecimal code with the goal of using it to write a cool program "someday".

Code: Select all

from	LunaVorax <[email protected]>
to	Caitlin Shaw <[email protected]>
date	Sat, Apr 3, 2010 at 7:07 AM
subject	Re: CloneKeen development process


Great ! Thank you for all this information. I just forgot to correct you in the precedent mail, Rayman 1 is a pure 2D game. Also, it was indeed available for the Playstation, but I'm working on the PC version which is DOS based.

However, since I own a Playstation copy of the game and since I already found some documentation on the Playstation hardware too (and I actually know a crazy coder who code a bit for this platform) I may also work on the Playstation version since I believe the resources of both version of the game, not being so different.
That was probably the longest sentence I ever made, I hope I wrote it correctly.

As for the HEX Editor part, I know that kind of tool is widely used in general hacking, but I don't remember seeing any informations (I mean "in clear" of course) and I think no match were found between two files of the same extension.
I'll have to check that today, when the sun will shine (it's 7am). Also I'll have to check if the files are the same between the PC and PSX version (maybe the PSX version use the same technology but is a bit more compressed).

Again, thank you very much for your efficiency for that answer. I'll mail you again as soon as I have other questions or other news.
LunaVorax

Code: Select all

from	Caitlin Shaw <[email protected]>
to	LunaVorax <[email protected]>
date	Sun, Apr 4, 2010 at 3:03 AM
subject	Re: CloneKeen development process



I had no idea it was a PC game. When you emailed me at first I glanced at some screenshots to get an idea what we were talking about and from my glance at those it looked 2.5D. But I see now that it is not.

Looking at the files of the installed game, it seems hard to locate much...there are many files claiming to be maps (.LVL or .WLD extension) and I would say that at least most of them probably are. I notice those files seem to contain an inoridinate number of 0x8F bytes--I would hypothesize that this byte might represent the background, or "no tile"; and based on the files' appearances in a hex editor and their size, the level data may even be just raw uncompressed tiles (there are lots of runs of a single byte usually 8F, then sparse patterns which repeat at intervals).

I'm not sure where the graphics are; my best guess is the file VIGNET.DAT, which seems to be an abbreviation for "vignette". I fear it may just be related to cutscenes though. I tried renaming this file to simply "v" so that the game could not find it and launched the program; it went through initilization fine but when it got to the intro screen it was just blank for about 3 seconds, then crashed with an error that it could not find VIGNET.DAT. The error message included the word "pcx", so I hypothezied that VIGNET.DAT may be a modified version of the PCX file format, which was common when Rayman was released.

All PCX files start with 0A (the magick), then a byte between 0-5 (the version), and then byte 01 (to specify RLE compression, the only compression type for PCX's). Following that is the BitsPerPixel, which can have only a limited range of valid values. In VIGNET.DAT I found a single run of the bytes 0A 05 01 08, indicating the start of a 256-color PCX file. Deleting everything before these bytes and resaving with PCX extension allowed me to open the image in Gimp, which turned out to be, I think, one of the background images. The palette was wrong, but it was definitaly a real image.

I couldn't locate any other images as of yet though. I may take another look at it later.

Code: Select all

from	LunaVorax <[email protected]>
to	Caitlin Shaw <[email protected]>
date	Mon, Apr 5, 2010 at 5:59 PM
subject	Re: CloneKeen development process


Wow, I didn't thought you would help me start looking into theses files. This is very kind of you, I really appreciate it.

If you don't mind, I'll post the mail we shared on the forum to share the information and hope creating a tiny snowball effect as some people may try different searches with the game. I'm just waiting for your approval.

I don't have that specific knowledge of PCX files, but I've should have thought about it since, yes, PCX were common at this time and we also find this format in the Windows installation of the game.

However, despite your very clear and precious instructions I was unable to find the string 0A 05 01 08 in the VIGNET.DAT of a later version of the game called Rayman Gold/Rayman For Ever (both have the same VIGNET.DAT file since the md5 checksum is the same : 878ad62b44b56f000be96ec87b4b3fe8). The working version of the file has the md5 checksum edc45209de8c770d8f272a2bacecce25.

Anyway, the image you've found is indeed a background of the game as you can see it here http://image.jeuxvideo.com/images/ps/r/a/rayman-playstation-ps1-054.jpg, but more of 2mo for such a file seems a little bit too big, and I wouldn't be surprised it contains a bunch of other files/information. Also, the amount of information deleted before the 0A 05 01 08 seems to be pretty big not to be interesting.

I'll start searching for other files that may contain indentical information.

Thank you very much for your help !
LunaVorax

Code: Select all

from	Caitlin Shaw <[email protected]>
to	LunaVorax <[email protected]>
date	Sat, Apr 10, 2010 at 12:21 AM
subject	Re: CloneKeen development process




Please feel welcome to share our mail on the forums if you think it might be useful to anyone.

The file I found the image in does indeed have a MD5 of edc45209de8c770d8f272a2bacecce25 like you said. I'm quite certain though that there is other information in there and that image I found is probably not of great significance to the game since it is the only one in raw pcx format.

You'll notice under PCMAP there are directories for "IMAGE" and "MUSIC", although the contents of those directories are ".LEV" files just like all the others directories; seeming to indicate that they are maps. I suspect the extensions may be lying, and perhaps the graphics can be found in there, in some unknown format. Possibly a modification on PCX.

I don't know how many levels or worlds Rayman 1 has, so not sure if the 13 files in IMAGE could possibly account for all the level graphics.

If I was going to pursue it further, I would probably get some sort of DOS debugger set up and look at what files the program is opening at different points, to get a better idea of what is important to what. Another thing you can also try is intentionally corrupting or swapping files (e.g RAY1.LEV with RAY2.LEV), and watching what gets messed up.