'Too many items' bug

[Kanhef]

Has anyone figured out exactly what causes this bug? Is there a predictable, reliable way to trigger it? I've seen several theories – too many objects on a map, too many items with scripts attached, too many Pip-Boy entries – but it's not clear which one is right.

I want to get some savegames from right before and after the bug has been triggered, to try and find out what exactly gets corrupted and if it can be repaired.

 

[Glovz]

I believe killap's figured this out for RP 1.3 - I believe this issue deals with items that have scripts and having too many of those items on the same map or in the players inventory.

Hope I got that right and I hope that helps.

 

[Misteryo]

Killap posted this on pg 103 of the Restoration Project thread:

Will the 1.3 Unofficial Patch also require starting a new game?

Yes and no. If you want the "too many items" bug fix to take effect, you will have to start a new game. If not, then you can continue right along with an old save. With the RP 1.3, if you try and use a 1.0/1.1/1.2 save, the game will crash when you try loading it.

If you read backwards from that page, you should be able to find some details.

misteryo

 

[Darek]

It happens only when you try to leave a map bringing with you too many items that have scripts attached to them. My tests showed that the more scripts that was on a map (all scripts, not only items), the fewer script items I needed to bring out with me to corrupt the map. I did list most, if not all those items before.

You can save your game while you are on the map with all the items without causing any corruption. It happens only when you leave the map and the game saves the state of it.

While the game is running it uses temp files (in the Map folder), later when you save your game those gets compressed (gzip) and put in your saveslot. Normally you can rename these and open them in the mapper (the temp files or decompress saves) , but the corrupted map always crashes the mapper too. So I had no luck finding anything there.

killap removed the scripts from all the holodisks and from a few non-quest related items, and put everything into the Obj_dude script. I had a test run with this and was unable to cause any map corruptions. Admittedly I didn't test it extensively.
He also removed the brahmin shit from San Fran (they had scripts too, and they had an effect in this matter).

I don't have a save for you at the moment but can put one up later, just tell me what F2 version/mod you want it for.

Alternatively, if you have the mapper, just place these items on artemple.map, start a new game, pick up the stuff, then go into the temple and back out. Voila, you get your crash.
You don't need all those items though. IIRC 17 or 18 was enough to crash that map.

 

[Per]

Is it known how the black button bug relates to this?

 

[Timeslip]

Is it known how the black button bug relates to this?

That's a separate issue caused by having too many entries (quests + holodiscs, iirc) in your pipboy. It's fixed by sfall, whereas the too many items bug is not.

 

[Kanhef]

Thanks, that should be enough to work with. Crashing the mapper won't be a problem – a hex editor is my tool of choice.

 

[Darek]

Is it known how the black button bug relates to this?

The only connection is that the holodisks are involved in both of the bugs, or rather having many holodisks. As the bugs usually hits around the same time as well, it's easy to see why they were thought to be related.
Oh, and it's location + holodisk entries in the Pip Boy status screen.
23 for 1.02 patch and 22 for the unpatched version.

@ Kanhef, sometimes you may have to run back and forth twice between maps for it to crash. I never did figure out when/why that is.
It may be that if you have fewer items, it needs to "build up" the crash, but when you have many it crashes right away. Meh, I don't know.

Here is a save before the corruption, and here is one save after. That's with the official patch only.

Or if you just want the extracted maps, here's the uncorrupted one, and here's the corrupted one.

Good luck. I hope you can find something.

 

[Darek]

I compared my working and my broken ARTEMPLE.MAP and these are the differences:

Line             Hex                normal       corrupt

00000020         23                   14           0F
00000030         3B                   DD           ED

00000120 - 00000130 20 bytes missing from corrupt map: add 00 04 9E 9A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 with start at hex 128 to compare easier, or take off 20 hexes to the ones I've given.


00009d90         9D8B                 18           02         Hex 9D77 on corrupt map.    This is the one crashing the map!
00009e10         9E17                 05           1D
00009e50         9E57                 05           1D
00009e90         9E97                 05           1D
00009ed0         9ED7                 05           1D
00009f10         9F17                 05           1D
00009f50         9F57                 05           1D
00009f90         9F97                 05           1D
00009fd0         9FD7                 05           1D
0000a010         A017                 05           1D
0000a050         A057                 05           1D
0000a090         A097                 05           1D
0000a0d0         A0D7                 05           1D
0000a110         A117                 05           1D
0000a150         A157                 05           1D
0000a190         A19F                 05           1D
0000a1d0         A1DF                 05           1D
0000a210         A21F                 05           1D
0000a250         A25F                 05           1D
0000a290         A29F                 05           1D
0000a2d0         A2DF                 05           1D
0000a310         A31F                 05           1D
0000a350         A35F                 05           1D
0000a590         A59F                 02           01
0000aa00         AA0B                 02           01
00013560        1356F                 70           90
00013570        13573                 01           49
000140c0        140C7                 10           30
                140CB                 1A           62
00014860        14867                 50           70
                1486B                 1A           62
00014ca0        14CA7                 90           B0
                14CAB                 1A           62

Only one hex matters to crash the game. Changing that makes the map playable again. What the other differences are/mean I don't know. As far as playing the game I could not notice any difference. I did not change anything on that map between the saves. Maybe player position, time, light? Any idea what it could be and if it's important?
Maybe I should try to change the hex value on a different working map (if I can find it), and see what happens.
Edit: I think the missing 20 bytes is the player not being on the map, if so that's not part of the bug.

 

[Kanhef]

Almost everything in a 4-byte value, so I'm referring to them by the base address.

0x38 is the current time. I don't see why that should be part of the map file, but that's what it is.

0x20 is the number of local variables. There aren't any in the original artemple.map in master.dat, (no artemple.gam either) so they must be set by scripts. This decreases by 5 in the corrupt file, which is why the 20 bytes at 0x128:0x13C were removed; they were the variables that were deleted. Without knowing what scripts are in use, I can't tell what's happening.

0x9D88 - the part that causes the crash - is the fourth value after the end of the tile list. This is the start of the script section, which is unfortunately not well documented. The section 0x9D8C:0xA660 contains all the 5->0x1D changes, and appears to consist of multiple records with lengths from 0x40 to 0x4C. There are 35, so the problem byte isn't a count of them. I'll need to pick it apart to find out more.

0xA660:0xAA08 is a block of data, I have no idea what it's for. The remainder of the file is all the objects on the map.

 

[Darek]

Almost everything in a 4-byte value, so I'm referring to them by the base address.

I'm having a hard time understanding this as it is. Your numbers don't mean anything to me. Oh well, I guess I will have to read up on a few things.

0x20 is the number of local variables. There aren't any in the original artemple.map in master.dat, (no artemple.gam either) so they must be set by scripts. This decreases by 5 in the corrupt file, which is why the 20 bytes at 0x128:0x13C were removed; they were the variables that were deleted. Without knowing what scripts are in use, I can't tell what's happening.

The obj_dude script have 5 Lvars, so that's probably it. That's as it should be then.

0x9D88 - the part that causes the crash - is the fourth value after the end of the tile list. This is the start of the script section, which is unfortunately not well documentedThe section 0x9D8C:0xA660 contains all the 5->0x1D changes, and appears to consist of multiple records with lengths from 0x40 to 0x4C. There are 35, so the problem byte isn't a count of them. I'll need to pick it apart to find out more.

That the problem is in the script section makes sense. I played around a bit with it's value, 11-20 is fine, higher or lower than that crashes the map.

I have a feeling that I am missing something in my tests. I tried to open the corrupt map with Vad's save editor that can edit Mvars and Lvars, but it could not open the map save, it couldn't even open the the map from the working save. So either the editor just can't handle the extra items I added to the map, or the map got partly corrupt already when I stepped into the map with script items.
That may mean that my "working" maps may not be the best to work with. Should maybe go into the map without any items, and then out, for best comparison

I crashed the Arcaves.map as well and then tried to find something there. Unfortunately it's way bigger, with thousands of differences between the corrupt and working map. Well, not really, its a few lines of 8*16 bytes that are skewering up the comparison (at least most of them were that size, if not all). I added the lines that were missing in the corrupt map and removed the ones it had that were not in the working map. That left some 60 odd differences. I found a byte value that if changed would make the map playable again, but it would only work on the map where I had changed the lines, not on a map where I only changed that value. So I don't know how useful that is. Any clue what it is?

Line               Hex              working     corrupt
00000020           23                 BF           BA

Anyway, I can produce any corrupt maps you may want, if you feel like looking at more of them. I would assume smaller, single level maps are easier to work with.

EDIT:
I'm so confused to how this whole bug works. I tried braking a few more maps, and it seems like the crash bug only happens if you enter a map with too many items and then leave again with too many items. If you acquire the items on a map it don't seem to corrupt when you leave. But Vad's save editor can no longer open the map saves, so something happens.
Also, I can't figure out why I need to enter/exit the ARCAVES.map twice to cause the crash bug.

 

[Josan12]

It's great that you guys are tackling this as it's looking like there might be a significant increase in items with scripts attached for future updates of the RP and MM.

I hope you make progress with this!

 

[Per]

Are there a lot of other behind-the-scenes things you've found out exactly how they work? The Jet Antidote bug? The Jinxedes? Night Vision? The pariah dog bug? Sneak and steal?

 

[Kanhef]

http://falloutmods.wikia.com/wiki/MAP_File_Format is somewhat helpful.

Sorry if my notation is confusing. As an example, the number of local variables is stored as a 32-bit integer, occupying bytes 0x20, 21, 22, and 23. Only the last byte is non-zero, but the actual value starts at 0x20, not 0x23.

 

[Darek]

@ Per, I have no idea about any of those, I don't even know who your question were directed to.

@ Kanhef, here is a clean set of ARTEMPLE.MAP.
Unfortunately the old "normal" map were tainted by the first stage of the items bug. It also had the obj_dude script and a locker where I placed the items. None of that this time.

I think the problem byte is counting the scripts. It says 02 and there are 3 scripts (but only two different ones). I added two new scripts to the map and the value changed to 04, and they were also written into the script section. I believe every script is 64 bytes long.

I think that the script section have reserved space for just over a 1000 bytes for the scripts. At least that section is cluttered with symbols that gets overwritten every time a new script is added.

The scripts attached to the items gets added to the section just like the normal scripts, but they do not get removed from there when the items are removed from the map. I tried this with 10 items. I don't think they are supposed to get removed. Instead the problem byte is supposed to change. With 10 items the value changed to 0C witch equals 12 (I had to look it up ). That fits if the Obj_dude script is not counted, which it probably isn't.

I mentioned earlier that there sometimes are two stages of the bug.
That was what happened with the map I put up earlier. The first time I left the map the script number byte stayed at 18 (it was supposed to go down to 02). The second time I left it, it correctly went down to 02, which is when the map will crash the next time it is entered. So the script number byte sometimes not going down correctly is actually a good thing, you get a second chance.
Anyway, the value of it is not the problem, something else is.

The number of scripts in the script section is not what sets that value (since they never get removed), only how many scripts are currently on a map decides that. I think the problem is that when the script section gets too large (in the corrupt map it was twice the reserved space), the game can no longer handle the movement of scripts correctly. It tells the script number byte that the scripts have been removed from the map (like they should be) when in fact they haven't. "Stage one" (happens only sometimes) seems to be even more fucked up since it neither removes the scrips nor change the script number byte, but at least that doesn't crash the game.


I just want to say that the above contains a lot of speculations and that I'm not really sure about much of it, and since I have pretty much only tested much with ARTEMPLE.MAP I don't know how much will be the same with the other maps.

Also, thanks for the link, I'll have a look at it.

 

[vad]

I tried to open the corrupt map with Vad's save editor that can edit Mvars and Lvars, but it could not open the map save, it couldn't even open the the map from the working save.

There are two reason:
1. Bug in my editor.
2. Map was already corrupted.

I will try to tell everything what I know about script section.
It consist of 5(not absolutely sure) sequences of scripts.(Items, critters, etc)
Each sequence consist of:
1. Number of scripts used in this sequence. 4 byte
2. Blocks of scripts. Each with 16 scripts.
After block there are always number of scripts used in this block.( = 16 except last block)
If all scripts in block are used (previous number = 16) then there are 4 unknown bytes before next block.

Scripts records have different size in different sequences. They are not always 64 byte!

00009d90 9D8B 18 02 Hex 9D77 on corrupt map.

This was number of scripts used in 4th sequence. It became 2 but it seems that number of blocks with scripts records in this sequence remain the same. And I think that Fallout can not load this map because It looks like there are 1 block in sequence, but actually there are 2 of them.

 

[Darek]

00009d90 9D8B 18 02 Hex 9D77 on corrupt map.

This was number of scripts used in 4th sequence. It became 2 but it seems that number of blocks with scripts records in this sequence remain the same. And I think that Fallout can not load this map because It looks like there are 1 block in sequence, but actually there are 2 of them.

That makes sense I suppose. Since the value had to be between 11-20 (17-32) for the map to work. So that value is only important to tell how many blocks there are, the exact number of scripts are irrelevant?
What I would like to know is how it tells what scripts are on a map, because that seems to be where it goes wrong.

Thanks for looking at it. Unfortunately I don't understand most of this.
I might post some questions later...

 

[Per]

@ Per, I have no idea about any of those, I don't even know who your question were directed to.

To the amorphous mass of finding-things-out people to which your apparent knowledge of the too many items bug marks you as belonging!

 

[Darek]

I've been thinking and put up some pictures to show what I mean.
I removed a few bytes before the script section so I could look at it easier (only for the pics).
The blue spot marks the number of scripts byte.

Pic 1: working map, no extra scripts.

1. These are the scenery scripts on the map, two for fire scenery.
(they look like 64 bytes each to me, but I may be misunderstanding)

2.This space, I think, is reserved for 16 scripts (or 14 since there are already two on the map).
At least when more scripts are added to the map they get added there (2 lines for every script), replacing whatever garbage is there now, until it reaches 16. It can change a few values, but it never removes those lines once they are put in there. It doesn't matter that the scripts are only temporarily on the map.

Below all the garbage, the critter scripts section starts.

Pic 2: Bugged map.

1. Two scripts, most likely the scenery ones that are on the map to begin with.

2. 16 scripts in a block.

3. When the first block of 16 scripts get full and there are more scripts to add, they get added like this.
But normally they get put AFTER the first block of 16 (from what I've seen). Here I don't know what's going on, and I think it's supposed to be 16, which is not the case here.

On a map with 18 scripts total, and not bugged, it kept the the 16 first scripts (like fig.2 but in the place where fig.1 is) and then added 16 more looking like in fig.3 after the first block. Don't know why it adds 16 more, should have been enough with two. Maye the game likes to be prepared? That extra block of scripts only exist on a save where the player are on a map with the script items. When the player leaves that map, taking the scripts with him, that extra block of scripts gets removed.

So I'm thinking what is causing the crash is that the extra script block gets placed in the wrong place.
It tries to delete that script block but fails cause it's not allowed to (that space is reserved for scripts, can be edited but not deleted). Even though it fails to remove the scripts it still changes the value of the script byte, so that the next time the map is loaded it's looking for only one script block when in fact there are two, and therefore crashes.

I tried to delete the whole section in fig.3 and the two last scripts from fig.2, that made the map playable again.

Again, this is only tested with the first map and I have not looked how maps with more scripts to begin with looks like.
Edit: Changed some stuff that was totally wrong.

 

[vad]

So that value is only important to tell how many blocks there are, the exact number of scripts are irrelevant?

Exact number is also need to know how many records in last block are used. Number after block is not reliable. Unused records in last block are filled by junk and may contain absolutely random numbers.

What I would like to know is how it tells what scripts are on a map, because that seems to be where it goes wrong.

When you add some script to some object, Fallout creates new script record in appropriate sequence (replaces junk record or adds new block). If you add one script to two objects, two new records will be created. Each script record has unique for this map identifier and Fallout writes this identifier to object record at offset 0x0040.
Since each sequence is used for definite type of object, if you sum records from all sequences that should be all scripts which are present on the map.

Information from TeamX about scripts:

Script type   Type(from SeaWolf)    Type(from Mapper2.exe)    Size of script descriptor
0                     -                        s_system                     ??
1                     spatial               s_spatial                     72
2                     items                  s_time                        68
3                     scenery            s_item                           64
4                     critters               s_critter                     64

Script types 0 and 2 were never meet.