Anyone working on a character editor yet?

[skidz]

Hey Ayeso, what software do you use to monitor files for changes?
Also, what language do you use for writing your neat looking apps? I use C# and .net myself.

Great work btw.

 

[Egon_Freeman]

I also saved at the moment i leveled... then added my skill points, resaved right away and WOW the file structure is basically 99% different... almost EVERYTHING in the file is changed. so.. im not sure what to do now.

WARNING - WARNING - Brain Dump ahead! (for those of You who suffer from the "tl;dr" syndrome - that means skip me and wait for some interesting updates instead - You have been warned)

Now I am just saying here it may be that I am completely off-track and uber-wrong about every single thing I write here - but it doesn't hurt to experiment, now does it?

Well, with that out of our systems - by all means, let's roll.


Forget Fallout save games. Think `Oblivion`. "Fixed save-game positions" mean nothing in the world of Gamebryo. What You really need to figure out here is the mechanism used to save the games.

(for Your viewing pleasure, I've enclosed my ramblings in SPOILER tags so You don't have to read it if You don't want to)
[spoiler:2112822183]I've had troubles like these with a different game - Hellgate: London (oh goody, I remembered the name)... Each time I wanted to compare saves, they were vewy vewy diffwent. The reason behind this was simple: the game used dynamic structures, in save file as well as in memory.

I must admit, adding items through a memory editor was one hellish piece of action (wink wink). I remember needing to expand inventory area, make the game think it had extra mem space for it, and then inserting appropriate structures (item descriptors). In the end it got so tedious, that I stuck with hunting down item amounts and changing that (which was also a huge pain in the rear - yet another wink ;P). Not to mention that in the end you had about 4-5 item IDs, all of which had to be changed for the game to recognize it, and sometimes it didn't anyway...

...oh and did I mention saving and reloading immediately? so the game wouldn't corrupt its own memory and crash without noting the changes, AND sometimes removing both the erroneous item AND the item it was based on... in essence, trying to mod say, ultra-rare (legendary) minerals and failing (items of one type could be exchanged for items of the same/similar type) meant losing those minerals altogether - and try hunting them down for like half a game to get, like, 10! :/ But I'm rambling, sorry.

Anyway [/OT], I figure something similar is happening here.[/spoiler:2112822183]

It's a hell of a lot easier to just dump structures to a save file and then somewhat compress or encode it (no, not to prevent tinkering as some might think - just to optimize it for access). Therefore -- no signposts for You, I'm afraid.

But there is yet hope -- Oblivion editors? (humongous nudge?). As I mentioned earlier - think Oblivion.


And just to spoil the fun, here's what I'd do:

#1) Get Oblivion, extract relevant data files (item IDs, quests, character stats, the works) and compare that extensively with Fallout 3's stuff.

#2) After that, fire up Oblivion, make a few saves and observe how the editor trails through those. (I do seem to remember there WAS an editor... ;P) Contact the editor author if required.

#3) Pray, pray some more, and then pray some more still that significant changes were not made to the saving system. You'll want to find similarities in saves, based on what You know - spells are perks, skills are still skills, and basic attributes are SPECIAL stats.

#4) Vewy impowtant: Oblivion used a different way of calculating abilities/skills (skill training and leveling up that particular skill only). But the engine is kind of on Your side this time -- all values are calculated in-game (this time to prevent tinkering ;P).. That means two things for You:

a) on the negative side -- You'll have to do Your own calculation in the editor to show accurate changes that would show up in-game (direct editing of say, SNEAK - won't do, as it's calculated from Agility with skill points and god knows what else), aaaand

b) on the positive side -- You're hunting for a limited number of known values and flags - seeing as all other values are calculated in-game

Now that I think about it, it could also mean that on each level-up there's an additional overlay-layer of level-up data stacked for accurate calculations, but let's not go there......... (the existence of the "Tag!" perk would suggest no such thing, so let us pray once more to the mighty Atom...)



Worst case scenario if You f-up? The game boots Your savegame to default character, default inventory, default stats/special/perks, default quests, outside Vault 101 for the 1st time... the usual.

[Edit]
Since we're talking epic drama here (no kidding!), why not entertain the notion of big matrices that combine to form complex variables in-game? But that's seriously scary stuff, man! I mean, get an ef-ing degree in rocket science just to edit savegames...?
[/Edit]

Please don't stone me.

 

[Forcepoint]

Hi Ayeso,

I'm working on a savegame manager for FO3, and it needs to read data out of save files. I'm trying to decipher the file's format and it appears it's very different than the Oblivion format.

I'm posting my discoveries on this wiki page, and I'd like to ask if you could do that too.

Thanks and good luck on the editor

 

[Grim-Ascension]

Just so you guys know, 'GetXPForNextLevel' or whatever, doesn't advance you one level, it just shows you how much EXP you need to level. The proper command is 'advlevel' and you'll level up.

 

[Egon_Freeman]

Okay, so I looked at the save files. Yeah, for real ;-P

Once again, a foreword: I may sound like "Mr. Obvious" - sure. But I haven't noticed any notes on these topics, and it also helps to know what things aren't, if for no other reason that simple elimination.


The first thing I noticed is a seemingly obvious thing -- it's one save file. One. If you recall in Fo2 or Half-Life and such, a save file consisted of data files for several locations, besides the obvious .sav file for the character stats. These location data entries, if for nothing else, could be used to store object states like doors, safes, computer terminals and such. This possibility opens up new ideas for research: try comparing a savegame before and after cracking a safe? In the console, take note of the safe's ID tag, and see if it exists in the savegame? In the game it is also possible to relate to obejcts by their IDs (by putting "ID." before the action, player.showinventory works as well as 14.showinventory as far as I know) - so perhaps a quick reference or two to INTs suspected of being object codes? If this would prove true, a potential elimination of all alike entries (discarding of world information) could somewhat limit the search area.


Now, all that data could be saved as a really big matrix of items in the game, sure... it wouldn't make much sense, but it's possible. But no, I'm not so sure.

One way to determine if this is even remotely possible is to check the size of the file before and after discovering a new location. If it suddenly got a whopping increase - yup, it's storing locations.

Another (related? maybe) thing that got me interested is this repeating pattern of HEX, that looks like a marker. I'm talking about the combination of 7C's, 7F's and FF's.

Sample: 7C FF FF 7F 7F FF FF 7F 7F FF FF 7F 7F 7C

Another thing that intrigues me, is the existence of the following keywords at the later stages of the savegame: Open, Close, SpecialIdle.


The "visual" structure of the savegame changes in a pre-determined order.

Also, a note on the bytes. It would appear that the save-game data is stored in little-endian (low-order byte first). Multiple values are separated by the | character (7C in hex). A quick shot would be seen with the character name - Ahleena being 7 characters long, and directly before it having 07 00 (that is 00 07 in little-endian). My characters work the same - Shodan has 06 00 7C NAME-HERE, and Miyu has 04 00 7C NAME-HERE. Better keep the little fact in mind, heheh

Somewhere in the 2/4th part of the savegame we come across "FFields", as I call them - vast fields of irradiated FF FFs. These are most likely placeholders. What they're supposed to hold is to be determined, but placeholders have this knack of keeping space for things of pre-determined size, and sometimes order. This is a 'hot-zone' for me - keep an eye out for changes in this area, between savegames.


--------[ like, 20 mins later ]--------

Okay. So I loaded it up, and it is certain that things get dumped in a very weird way. For example, during the game, the quick-save isn't "truly saved" - it's dumped into quicksave.fos.tmp, a file which is somewhat more messy. After that it gets put to order in the final savegame.

As I suspected, strings are saved as 7C uint_16 strlen (little-endian) 7C string 7C. So, for example "Close" would be saved as

7C 05 00 7C 43 6C 6F 73 65 7C
(which would correspond to "|..|Close|" on char display ;P)

This reinforces my theory that the | "posts" (heck, they look like posts) are in fact "posts" separating data -- that is, markers. After checking up on location names and karma stats, I have also confirmed the little-endiannes of integers in the save files - now I am SURE. There is only ONE way 0C 00 would translate into 12 (the length of "Vault Guardian").

Here's my take on decoding the b**ch: separate the whole file by 7C - anything 2B is a 16bit int, anything 4B is a 32bit int, 2B followed by some form of readable character is a string with length. Anything else is unknown data.

Also, separate the whole file by the marker provided above, and further split by 7C. Maybe that way we could make some sense out of this seemingly random data we have.


And oh, one last note -- something tells me that we're dealing with a "pack" of sorts - a structured entry list, sort of like a database file, or a resource file. What I mean by that is that we're looking at smaller "file-like" entities packed together. It may be worthwile to try and determine if the file could possibly be "de-indexed" into smaller chunks by some sort of key-like structure (directory?). If it's anywhere, it's in the file; and if so, it's in a prominent location (beginning? end - more like it?).

I also noted that the file's structure changes in a pre-determined basis - I was wrong, obviously. But I've also noticed that I was still dealing with the same "parts" of the file, only in a slightly different order. So, the idea of the file being somehow indexed could really lead to something (how otherwise would the game keep track of the contents if they're shuffled up like that?)...


I hope this helps somewhat.


--------[ like, 15 more mins after that ]--------
I took a second glance at the savegames - the two that were both close and far apart (61 seconds, 1 perk and 25? skill points apart, to be semi-exact ;P). I have noticed that some parts of the savegame did NOT change - I repeat, they did NOT. They only appear different because they have been MOVED. Where there was, say AAABBBBCCCCC, now is AAADBBBEECCCCC, and it looks different. I have noticed several new data entries (whose meaning of course eludes me ;P) that I think are (or could be) related to the perk addition. The point is, changes were made in three consecutive places - so I'm guessing ID entry, STATUS entry and CONTEXT entry. Or something. An interesting fact: I used the retinal scan algorythm to compare the two files, and the seemingly added entries, when summed up, seem to be roughly of the size that has been added to the savegame overall... we don't know WHAT it is, but we certainly can track its movement - so to speak... (writing of an app that would track additions seems inevitable now)

Still... It would seem that my little scary thought of having layers upon layers of accurate complexity in the savefile just got some meat on its bones... :/

--------[ The end for now ;P ]--------


Oh, and some random head-up-my-ass-ness. Skip for health.
[spoiler:e6a32a0f70]Moving on (r. I had a look at that Wiki page. Word of advice: strings... are meaningless. As far as I can tell, these are just what they appear to be - written text. Instead of "Moriarty's Saloon" I could write "My very own ass" (with appropriate corrections) and it would still load the same, and I don't think it would take me to the sacred place... Now don't get me wrong, it's a great idea to collaborate using a Wiki like that, but You're approaching this matter from the very same sacred end of things. You don't go hunting for random INTs and try to assign meaning to them... You go hunting for patterns.

Although I am absolutely with You on this one - if it helps to hunt around like that, I'll officially take my own head out of my sacred place.

And before You kill me, chop me up and hang me to dry in one of those raider locations we all oh so very much love and adore , ... uhh, okay. Do Your worst. I guess I didn't know what to say here anyway. [/spoiler:e6a32a0f70]

 

[Egon_Freeman]

And one important thing (double-post so that it would be visible) --


check other sites that're doing exploration into different areas. Oftentimes progress in those areas somehow inspires progress in SG editing -- like, someone for example finds the Master Item Status List, and some of the entries seem to look a loooot liiike those numbers I've seen here somewhere.... So, keep on top of it. G'night

 

[Forcepoint]

Wowz, that was a long read Anyway, I'll try to give an answer/comment on certain things;

Another (related? maybe) thing that got me interested is this repeating pattern of HEX, that looks like a marker. I'm talking about the combination of 7C's, 7F's and FF's.

Sample: 7C FF FF 7F 7F FF FF 7F 7F FF FF 7F 7F 7C

I've also noticed this, and I think this is a fairly good point to base the deciphering on.

Somewhere in the 2/4th part of the savegame we come across "FFields", as I call them - vast fields of irradiated FF FFs. These are most likely placeholders. What they're supposed to hold is to be determined, but placeholders have this knack of keeping space for things of pre-determined size, and sometimes order. This is a 'hot-zone' for me - keep an eye out for changes in this area, between savegames.

You may be right here. Haven't noticed this behaviour with the Oblivion files though. Then again... it's Fallout 3 - not Oblivion.

This reinforces my theory that the | "posts" (heck, they look like posts) are in fact "posts" separating data -- that is, markers. After checking up on location names and karma stats, I have also confirmed the little-endiannes of integers in the save files - now I am SURE. There is only ONE way 0C 00 would translate into 12 (the length of "Vault Guardian").

Here's my take on decoding the b**ch: separate the whole file by 7C - anything 2B is a 16bit int, anything 4B is a 32bit int, 2B followed by some form of readable character is a string with length. Anything else is unknown data.

Also, separate the whole file by the marker provided above, and further split by 7C. Maybe that way we could make some sense out of this seemingly random data we have.

This is really useful information, thanks Didn't notice this because I was looking at the savegames of just one character.

I also noted that the file's structure changes in a pre-determined basis - I was wrong, obviously. But I've also noticed that I was still dealing with the same "parts" of the file, only in a slightly different order. So, the idea of the file being somehow indexed could really lead to something (how otherwise would the game keep track of the contents if they're shuffled up like that?)...

I don't quite understand this part... Why would they make the order change if they could just do it the easy way?

I hope this helps somewhat.

Definitely does, no doubt

Still... It would seem that my little scary thought of having layers upon layers of accurate complexity in the savefile just got some meat on its bones... :/

Indeed... this might take a lot of 'investigation'

Word of advice: strings... are meaningless. As far as I can tell, these are just what they appear to be - written text. Instead of "Moriarty's Saloon" I could write "My very own ass" (with appropriate corrections) and it would still load the same, and I don't think it would take me to the sacred place... Now don't get me wrong, it's a great idea to collaborate using a Wiki like that, but You're approaching this matter from the very same sacred end of things. You don't go hunting for random INTs and try to assign meaning to them... You go hunting for patterns.

As far as I can tell I am hunting for patterns. In the File Header section (for example) you can see that... First you have the fileID, followed by 14 bytes, an int and another 2 bytes. 'Between' every row in the table there's a "post |". The Offset column is pretty useless and I'm going to remove that because that will almost always change if, for example, the PC's name is different. Maybe my vision of a pattern is totally wrong, but I think this is the way to go.

About the sacred place thingy... I know that changing the pcLocation value to "A body part that almost never sees daylight" won't make you magically go there ingame. It's not just for some reason that it's located in the Preview section of the save file Fallout 3 uses that value for the ingame saves browser to make you preview the save you might want to load (there's also a preview image in the ingame saves browser, haven't deciphered that yet though).

Also note that I'm trying to make the page look similar to the Oblivion save file format page at the UESP wiki, because that page was a great source of information and very readable.

And before You kill me, chop me up and hang me to dry in one of those raider locations we all oh so very much love and adore , ... uhh, okay. Do Your worst. I guess I didn't know what to say here anyway.

Hehe, will do

check other sites that're doing exploration into different areas. Oftentimes progress in those areas somehow inspires progress in SG editing -- like, someone for example finds the Master Item Status List, and some of the entries seem to look a loooot liiike those numbers I've seen here somewhere.... So, keep on top of it. G'night Smile

Do you know of any topics/pages/wutevah to keep track of??

Anyway, thanks for looking into this You've already given quite a lot of useful information.

~Forcepoint

 

[Egon_Freeman]

Do you know of any topics/pages/wutevah to keep track of??

http://www.bethsoft.com/bgsforums/index.php?showtopic=900266

this one, for example from what I can tell, they already made some modifications to the game's data files (like modding weapons, armor etc.)

And oh, here:

Why would they make the order change if they could just do it the easy way?

They DID go the easy way. If the data order changes in the file, it must be indexed. That way one could go about accessing the file like a regular directory, or a database. Believe You me, going about reading a 4.5 MiB file counting bytes is just wrong.

So... it doesn't really matter why or how the data order changes - if it's what I'm thinking it is, this actually makes our task that much easier. I mean, if they didn't care what order the data was in, it means we don't have to as well.

And yes, that is good.


---
One more positive side of this probable arrangement -- we can decode the data structure piece by piece, instead of having to decode the whole file at once. I mean, if they're indexed and not structured, that means they don't relate to each other in the file. Ergo, there may be some internal checks present (like embedded CRC), but other than that we can go around messing with parts of the save and concentrate on just that part alone.

That is, all of the above provided only if the file structure is what I think it is

 

[Egon_Freeman]

[Update]

As I have feared, most of the world data COULD BE indeed dumped into the file. I've written a small script to slice the file into manageable chunks (will post a link for the .php script somewhere, to run it you'd need a PHP interpreter)... here's the strings I've found (at least a part of 'em) :

[spoiler:233824c7f5]

(ELEMENT 4, LEN 4) Miyu
(ELEMENT 6, LEN 14) Vault Guardian
(ELEMENT 9, LEN 21) The Capital Wasteland
(ELEMENT 11, LEN 9) 001.05.50
(ELEMENT 2406, LEN 12) Fallout3.esm
(ELEMENT 2815, LEN 80) Sweet America. Hello again. This is President Eden, and I'd like to have a chat.
(ELEMENT 2817, LEN 91) Data\Sound\Voice\Fallout3.esm\MaleUniquePresident\RadioEncla_RadioEnclaveSpe_00045907_1.ogg
(ELEMENT 2833, LEN 91) Data\Sound\Voice\Fallout3.esm\MaleUniquePresident\RadioEncla_RadioEnclaveSpe_00074168_1.ogg
(ELEMENT 2847, LEN 93) It's time we discussed something rather important. The issue at hand is, well, my presidency.
(ELEMENT 2849, LEN 91) Data\Sound\Voice\Fallout3.esm\MaleUniquePresident\RadioEncla_RadioEnclaveSpe_00045903_1.ogg
(ELEMENT 2863, LEN 106) The question has been raised, I know, as to just how I came to be elected to this most illustrious office.
(ELEMENT 2865, LEN 91) Data\Sound\Voice\Fallout3.esm\MaleUniquePresident\RadioEncla_RadioEnclaveSpe_00045902_1.ogg
(ELEMENT 2879, LEN 45) Or, whether or not I had been elected at all!
(ELEMENT 2881, LEN 91) Data\Sound\Voice\Fallout3.esm\MaleUniquePresident\RadioEncla_RadioEnclaveSpe_00045901_1.ogg
(ELEMENT 2895, LEN 132) To that I must answer of course! Of course I was elected, sweet America! Isn't the right to vote the very foundation of a democracy?

[/spoiler:233824c7f5]

---[ Edit ]---
I just thought of something -- as it also stores filepaths, I now believe this section to be cache management. It could be either of the two, but there is an eyebot right outside Vault 101 (where the saves were taken)... it stands to reason, right? And is much more plausible. But! I am unsure (read below), this might still apply to the rest of the savegame...
---[ /Edit ]---



I was right on the spot with strings, though. Little-endian, and whenever there's a 2-byte chunk with a chunk of exactly the length as written in those two bytes immediately following them, it's a 99% string (there are a few that pop up from time to time that don't really resemble anything readable, but that's understandable, seeing as we're talking about version 0.0.0.0.1-very-alpha here ;P)


Also, I am very suspicious about those Open, Close, SpecialIdle and all that. They pop up quite oftenly, and all over the place. This leads me to believe these are entries for objects and their status definitions. For example, a character on the world map (say, a radroach or a bug) could be in the SpecialIdle state for saving, or until it sees something to attack? Or until the player (or anything 'attackable/interactable') enters into its range? Who knows. The rest of the strings are pretty much that.

I will post more I hope - I'm just in the beginning of actually reading the data files this script dumped on me


Here, grab a copy:

http://eter.sytes.net/parse_f3_save.phps

rename to .php after download, run with PHP parser like this:

php -f parse_f3_save.php

Please take note to have the file in the same directory as the script, and name the files quicksave_NUM.fos, and have exactly 3 (or change the number at the top of the script, where $files = 3; ) so I would have quicksave_1.fos quicksave_2.fos and quicksave_3.fos

the file will create 3 files:

full_structure_NUM.fos
strings_NUM.fos
last_bytes_NUM.fos

Strings are viewable as text, but for the other two I would recommend something like total commander's F3 (preview). Or for all of them. It somewhat structurizes the files into hex entries, so have fun


---[ Edit ]---
I would also like to confirm that the elements separated by 7C FF FF 7F... do indeed have a common structure! See for yourselves with the "parser".
---[ /Edit ]---

 

[Egon_Freeman]

So, anything new fellas? Anything found?

 

[Forcepoint]

No, not anything major yet. All the information I need for the savegame manager has been deciphered (Ingame name, level, karma etc.), so I'm just working on the manager and not on the rest of the file format.

On a side note, I've not yet tried your parser as I couldn't get a php parser set up on my pc Maybe you could host the parsed files somewhere and post the links here??

 

[Phnx]

How about a face importer/exporter? So we can start a complete new game with the same beloved face we created the game before?!

 

[Buzz2112]

Any word on this character editor?

 

[Dubby]

Anyone working on a character editor yet?

You're kidding right? Just use the console. *snicker*