Fallout DAT files

[Shadowbird]

I know this kinda weird right after my "save game format" request thread, but it's actually more urgent than that one.

Does anyone know or have the specification of Fallout master.dat and critter.dat file format? I've searched around, but I could only find the Fallout 2 format, which is different (for one, the file tree is at the end of the .dat, instead of beginning). There are several unpackers for Fallout, so the format must be known - I just can't seem to find if it's published anywhere. I also don't want to try to guess all the differences, so any help would be appreciated.

 

[Shadowbird]

OK, I've gotten through with Fallout DAT format, with the help of Fallout XentaxWiki and f1undat sources. I plan on putting up submitting somewhere a decent, nicely formatted HTML version with explanations etc., but who can tell when (if) that will happen, so here's the plain text rundown (additions and corrections are welcome, as always):

uint32	4	Directory Count
uint32	4	(unknown)
uint32	4	(unknown - always 0)
uint32	4	(unknown)

// Directory name Block - for each directory
 byte	1	Directory name length
 char	*	Directory name

// Directory content - for each directory
uint32	4	Number of files in the directory
uint32	4	(unknown)
uint32	4	(unknown)
uint32	4	(unknown)
	// File list block - for each file
	 byte	1	Filename length
	 char	*	Filename
	uint32	4	Attribute (0x20 / 32 == plain, 0x40 / 64 == compressed)
	uint32	4	Offset from the beginning of the DAT file, indicating the start of file data.
	uint32	4	Uncompressed file size
	uint32	4	Compressed file size (0 if equal to uncompressed)

// Data block
 byte	*	File data for all files

There's another thing I've been beating my head against the wall over, and I can't seem to get through - the compressed file format of Fallout 1 files (in the DAT). Is it the same as Fallout 2? If not, what is it/what are the differences? The uncompressing part in f1undat source is in assembler, which (me being a mediocre coder even in pascal and C++) is way beyond me.

 

[mvBarracuda]

Don't know if it helps but we've written a cleaner Fallout DAT1 / DAT2 implementation:
https://mirror1.cvsdude.com/trac/fife/engine/browser/HEAD/core/src/engine/maploaders/fallout

The files with a *1.cpp or *1.h are Fallout 1 specific, the ones with *2.cpp / *2.h are Fallout 2 specific. The other files are shared for both games.

 

[Shadowbird]

Well, I'm apparently not going to become a C++ coder - I couldn't quite follow the code you gave me and converting to pascal didn't work either. But I've finally managed to come up with a generic algorithm - so I humbly post it here and hope for any corrections or suggestions on part of people who can code much better than me (or just know the F1 file format better).

So here it is: Fallout 1 LZSS compressed files uncompression algorithm by Shadowbird
As far as I've tested, the result is 100% matching with files extracted with f1undat - both small text files and movies.

D = 4096; // Dictionary (a.k.a. sliding window / ring / buffer) size
All variables (FLAGS, N, O, L) initialized as 0.

REMEMBER! When writing to output below and after 2.2. (N > 0), write every byte also in dictionary, advancing dictionary offset by one. Once offset reaches the end of the dictionary, drop it back to the start (normally 0).

REMEMBER! Clearing dictionary means replacing it's contents with spaces (character #32, hex 0x20), not 0 or other value!

0. Set dictionary offset to D-18
1. If at the end of file, exit (duh).
2. Read N (2 bytes == word) from input. The absolute value of N is how many bytes of data to read (if N=0, exit, duh)
 2.1. If N < 0, read the absolute value of N bytes from input and write to output (do not put into dictionary). Go to 1.
 2.2. If N > 0, clear dictionary, repeat below until N bytes have been read from input, and then go to 1.
  2.2.1. Read FLAGS (1 byte) from input. Repeat below 8 times.
   2.2.1.1. If FLAGS *is not* a modulus of 2 (& 1 != 0), read 1 byte from input, write to output, then go to 2.2.1.3.
   2.2.1.2. If FLAGS *is* a modulus of 2 (& 1 == 0)
    2.2.1.2.1. Read O (1 byte) and L (1 byte) from input.
    2.2.1.2.2. Take away the High-nibble (first 4 bits) from L and prepend it to O;
    2.2.1.2.3. Read L+3 bytes from dictionary at offset O, and write them to the output.
   2.2.1.3. Divide FLAGS by 2, rounding down (>> 1).