sfall Arrays proposition
Am I the only one annoyed so much about how arrays work in sfall?
The problem is that there are 2 types of arrays:
1) "permanent" arrays which exist forever after they are created (unless implicitly freed), even between savegames (a bug in script code can lead to enormous save file size, and you can't easily know what arrays are "unused" to clean them out)
2) "temporary" arrays which exist for single frame and then deleted. They are useful in cases where you only need some small array in one function and don't want to bother about memory leaks. BUT, you can't use them in global script variable (which defined with "variable" keyword outside of procedure), which is really annoying.
HOWEVER, both array types are absolutely not something that regular programmer will understand as "arrays". By default, arrays in all other languages is a piece of information that is stored in memory. When you want array to be saved to file/savegame/somewhere else you usually use some kind of function for that. So you don't want "default" array type to be saved into savegames.
Temporary arrays are a bit different, because in many modern languages, there are things called ref counters, which can automatically free objects/arrays from memory after you don't need them (after last link is removed), but since we don't have such thing (and implementing one would be a pain), "temp_array" as it currently exist in sfall may be an option.
I was using a workaround for this problem in my own scripts:
- defined single procedure which creates permanent array and put it's ID in given sfall global variable
- I never use create_array directly, only use that procedure
But, it's not perfect. In some cases, when I need to pass my array between several scripts (like array of traps), this solution is ok (because I need to somehow pass variable ID between scripts and this is the only way since "imported" variables won't work with global/hook scripts). BUT, in other cases I only use array in one script.
Example:
- I have a list of some PIDs in mod's ini file.
- What I want to do is parse that list (text string) into an array of integers during script initialization (which happens after game load) and use that array efficiently throughout the lifespan of that script
- in this case I have to still store array and it's ID in savegame file, which is unnecessary, since I re-generate this array everytime and use it only in one script.
- what I want (and expect to be able) to do is use global script variable (not sfall global) like this:
Code:
variable pidList;
procedure start begin
if (...) then begin
pidList := create_array(10, 4);
// etc...
And not worry about savegames, because in real world arrays only automatically stored in memory.
In this case I don't need to worry about memory leak, because I will create this array only at script load, and I don't need to free it because it won't go to my savegame and will be freed from memory after game load.
I have a working implementation of third array type which don't automatically go into savegames in my current sfall build. But, there is several issues with that:
- To retain backward compatibility with older scripts, I will need to leave "create_array", "temp_array", "fix_array" functions as they were and add some kind of new methods. BUT, this will not fix the main problem with arrays (create_array will remain and will still be dangerous to use).
- I don't really want to add new opcodes to both sfall, compiler and decompiler, too...
- If my implementation will work differently from sfall master, I will have to apply my changes every time someone changes sfall
- While currently, RP don't use arrays at all, some other scripts might be using them in future and I will have a problem with compatibility.
So I propose to discuss how arrays would be implemented best to the point of including this into base sfall (if Timeslip approves).
I can do sfall coding/compiling myself (since I already have working implementation).
My current (not final) proposed course of actions:
1) create_array will always create "normal" array (not temp and not permanent)
2) temp_array will still create arrays that delete themselves at the end of the frame (I'm not sure about this, but at least you can expect something like this when it's called "temp")
3) (maybe) fix_array will turn normal array into permanent. This is easy to implement, but more logical thing to do would be to add functions like "set_sfall_array" and "get_sfall_array" that would work similarly to "get_sfall_global_int" and "set_sfall_global" (you don't "turn" array into permanent, you just do a one-time operation of storing normal array in the global sfall arrays storage; and when you "get_sfall_array" it will create a copy of stored array as a normal sfall array)
4) All existing mods/scripts that use arrays will have to be changed/recompiled. I know, this sounds terrible, but as I looked, there are not many mods that use them (RP doesn't use them, and from all mods I've seen only my and Jim's mod used them).
5) I can also add a ddraw.ini setting that will allow switching between old and new array behavior (for backward compatibility), but for all new mods it will be recommended to use new mode.
Additional ideas about other aspects of arrays:
1) Currently, we can store values of different types in arrays, but the downside is: you have to define number of bytes for all array elements. While it's normal thing for C programmer to do, we are talking high level language here and this is an implementation detail. In other words, scripter doesn't need to be bothered by how many bytes his value takes in memory (unless he wants to, to work with binary files for example).
2) While having array dereferencing operator is cool (shortcut to get_array/set_array), we also need new syntax to define constant arrays. Like this:
Code:
array1 := {5, 7, 8, 1, 2};
Could be translated like so:
Code:
array1 := create_array(5);
set_array(array1, 0, 5);
set_array(array1, 1, 7);
set_array(array1, 2, 8);
set_array(array1, 3, 1);
set_array(array1, 4, 2);
3) We really need some kind of maps, objects or associative arrays (with numeric or string keys). While this can be easily done even with current arrays, there can be no readable shortcut for this. Best we can do now is something like:
Code:
ASSOC(array1, "key", "value");
but what would be readable is something like this:
Code:
array1["key"] := "value";
I imagine it might be done by adding new function "create_map" which would create just another array but with some flag in it, telling sfall that this is an associative array. It will be stored differently and when you use "set_array" it will check, if it's a map, allow index of any type and value.
I will think more about this and when I'll see the whole picture, I will implement it.