sfall arrays (scripting) introduction

[phobos2077]

So I've been working to add some new features to sfall arrays for some time now (they will hopefully be in sfall 3.4). As I'm currently on a final stage (testing) I've decided to make a little introduction.

Reason for this thread:
1) introduce array concept to scripters who never used it. Answer any questions related to arrays usage.
2) possibly receive some feedback on my implementation from those who used them before

I already posted some rough explanation of what this is all about and what I want to improve here and here:

So what arrays are all about? Why do you want to use them?

Array is basically a container which can store variable number of values (elements). Each element in array can be of any type.
Arrays can be extremely useful for some more advanced scripting, in conjunction with loops.

Array elements are accessed by index or key. For example:

    // this code puts some string in array "list" at index 5:
    list[5] := "Value";

There are 2 different types of arrays currently available:
1) Lists - a set of values with specific size (number of elements), where all elements have numeric indexes starting from zero (0) up to array length minus one.
For example:

    // this creates list with 3 elements. Element "A" has index 0, element "B" has index 1, element "C" - 2
    list := ["A", "B", "C"];

Limitations:
- all indexes are numeric, starting from 0;
- to assign value to a specific index, you must first resize array to contain this index
(for example, if list is of size 3 (indexes from 0 to 2), you can't assign value to index 4 unless you change list size to 5 first).


2) Maps (or associative arrays) - a set of key=>value pairs, where all elements (values) are accessed by corresponding keys.
Differences from list:
- maps don't have specific size (to assign values, you don't need to resize array first);
- keys, just like values, can be of any type;

Both array types have their pros and cons and are suited for different tasks.

ARRAYS SYNTAX
​

Basically arrays are implemented using number of new operators (scripting functions). But for ease of use, there are some new syntax elements:

1) Accessing elements. Use square brackets:

    display_msg(arr[5]);
    mymap["price"] := 515.23;

2) Alternative accessing for maps. Use dot:

    display_msg(mymap.name);
    mymap.price := 232.23;

3) Array expressions. Create and fill arrays with just one expression:

    // create list with 5 values
    [5, 777, 0, 3.14, "Cool Value"]
    
    // create map:
    {5: "Five", "health": 50, "speed": 0.252}

NOTE: make sure to call "fix_array" if you want new array to be available in the next frame or "save_array" if you want to use it for a longer period
(see next section for details)

4) Iterating in loop. Use "foreach" key word like this:

    foreach item in myarray begin
        // this block is executed for each array element, where "item" contains current value on each step
    end
    
    // alternative syntax:
    foreach key: item in myarray begin
        // "key" will contain current key (or numeric index, for lists)
    end

STORING ARRAYS
​

Apart from lists/maps arrays are divided by how they are stored.
There a 3 types of arrays:

1) Temporary. They are created using temp_array function or when using array expressions.
Arrays of this type are auto-deleted at the end of the frame. So, for example, if you have a global script which runs at regular intervals,
where you create temp_array, it will not be available next time your global script is executed.

2) Permanent. They are created using "create_array" function or "fix_array" (from pre-existing temporary array).
This type of arrays are always available (by their ID) until you start a new game or load a saved game (at which point they are deleted).

3) Saved. If you want your array to really stay for a while, use function "save_array" to make any array "saved". However, they are, like permanent arrays,
"deleted" from memory when loading game. In order to use them properly, you must load them from the savegame using "load_array" whenever you want to use them.
Example:

    variable savedArray;
    procedure start begin
        if game_loaded then begin
            savedArray := load_array("traps");
        end else begin
            foreach trap in traps begin
                .... 
            end
        end
    end

PRACTICAL EXAMPLES
​

> Use arrays to implement variable-argument procedures:

    // define it
    procedure give_item(variable critter, variable pidList) begin
        foreach (pid: qty in pidList) begin
            give_pid_qty(critter, pid, qty);
        end
    end

    // call it:
    call give_item(dude_obj, {PID_SHOTGUN: 1, PID_SHOTGUN_SHELLS: 4, PID_STIMPAK: 3});

> Create arrays of arrays (collections) for advanced scripting:

   variable traps;
    procedure init_traps begin
        // just a quick example, there is a better way of doing it...
        traps := load_array("traps");
        if (traps == 0) then begin
            traps := [];
            save_array("traps", traps);
        end
        foreach k: v in traps begin
            traps[k] := load_array("trap_"+k); // each object is stored separately
        end
    end
    
    procedure add_trap(variable trapArray) begin
        variable index;
        index := len_array(traps);
        save_array("trap_"+k, trapArray);
        array_push(traps, trapArray);
    end
    
    // use them:
    foreach trap in traps begin
        if (self_elevation == trap["elev"] and tile_distance(self_tile, trap["tile"]) < trap["radius"]) then
            // kaboom!!!
        end
    end

Full operators reference:

Spoiler

*mixed means any type

> int create_array(int size, int nothing):
- creates permanent array (but not "saved")
- if size is >= 0, creates list with given size
- if size == -1, creates map (associative array)
- second argument is not used yet, just use 0
- returns arrayID (valid until array is deleted)

> int temp_array(int size, int nothing):
- works exactly like "create_array", only created array becomes "temporary"

> void fix_array(int arrayID):
- changes "temporary" array into "permanent" ("permanent" arrays are not automatically saved into savegames)

> void set_array(int arrayID, mixed key, mixed value):
- sets array value
- if used on list, "key" must be numeric and within valid index range (0..size-1)
- if used on map, key can be of any type
- to "unset" a value from map, just set it to zero (0)
- this works exactly like statement:
arrayID[key] := value;

> mixed get_array(int arrayID, mixed key):
- returns array value by key or index
- if key doesn't exist or index is not in valid range, returns 0
- works exactly like expression:
(arrayID[key])

> void resize_array(int arrayID, int size):
- changes array (list) size
- no effect on maps (associative arrays)

> void free_array(int arrayID):
- deletes any array
- if array was "saved", it will be removed from a savegame

> mixed scan_array(int arrayID, mixed value):
- searches for a first occurence of given value inside given array
- if value is found, returns it's index (for lists) or key (for maps)
- if value is not found, returns -1 (be careful, as -1 can be a valid key for a map)

> int len_array(int arrayID):
- returns number of elements or key=>value pairs in a given array
- if array is not found, returns -1 (can be used to check if given array exist)

> mixed array_key(int arrayID, int index):
- don't use it directly; it is generated by the compiler in foreach loops
- for lists, returns index back (no change)
- for maps, returns a key at the specified numeric index (don't rely on the order in which keys are stored though)
- can be checked if given array is associative or not, by using index (-1): 0 - array is list, 1 - array is map

> int arrayexpr(mixed key, mixed value):
- don't use it directly; it is used by compiler to create array expressions
- assigns value to a given key in an array, created by last "create_array" or "temp_array" call
- always returns 0

> void save_array(mixed key, int arrayID):
- arrayID is associated with given "key"
- array becomes permanent (if it was temporary) and "saved"
- key can be of any type (int, float or string)

> int load_array(mixed key):
- load array from savegame data by the same key provided in "save_array"
- arrayID is returned or zero (0) if none found

Backward compatibility notes

​

For those who used arrays in their mods before:
1) I've added new INI parameter "arraysBehavior". If set to 0, all scripts which used sfall arrays before should work. This basically changes that "create_array" will create permanent arrays which are "saved" by default and their ID is also permanent.
2) How savegame compatibility is handled?
Saved arrays are stored in sfallgv.sav file (in savegame) in new (more flexible) format, just after the old arrays. So basically, when you load older savegame, sfall will load arrays from old format and save them to new format on next game save. If you load savegame made with sfall 3.4 using sfall 3.3 (for example), game shouldn't crash, but all arrays will be lost.

Scripting library
​

To use arrays in all their might, some library functions may be necessary (like sorting, copying parts of array, etc.). I already posted some older versions of my libraries in this thread, so they will be updated some time after I've done with hard-coded stuff.

 

[JimTheDinosaur]

Hey @phobos2077 after slowly going insane I found out that the backwards compatibility thing for the arrays isn't working for me. At least they no longer get saved. I'm probably using it wrong tho, I tried just adding

arraysBehavior=0

to my ddraw, which apparently didn't work, and then stuff like adding the ddraw_adv.ini to the folder didn't do anything either... could you help me out?

edit: ah, I think it is cause I'm using the debug version and added it to the end of that...

edit2: yup, that was it, I'm a moron.

 

[phobos2077]

Has anyone used new arrays behavior in their mods yet (specifically save_array, load_array functions)? I'm refactoring this and will probably slightly change how arrays are stored in sfallgv.sav file so old arrays in savegames might be lost.

 

[Endocore]

This is great info, thanks. I'm wondering if there is any way to use arrays to make flexible lists. For example let's say we have something like:

list.msg
{100}{}{You see a }
{101}{}{cat.}
{102}{}{dog.}
{103}{}{bowling ball.}
{104}{}{marble.}

Let's say we want to have a response of line 100 plus a random selection from lines 101-104, but we want to repeat through the list and remove each item from consideration for further display after it has been used one time. The standard functionality display_msg(msg_string(100) + msg_string(random(101, 1004)) doesn't allow us to have that kind of control. Is this the kind of task for which an array might be helpful?

 

[phobos2077]

This is great info, thanks. I'm wondering if there is any way to use arrays to make flexible lists. For example let's say we have something like:

list.msg
{100}{}{You see a }
{101}{}{cat.}
{102}{}{dog.}
{103}{}{bowling ball.}
{104}{}{marble.}

Let's say we want to have a response of line 100 plus a random selection from lines 101-104, but we want to repeat through the list and remove each item from consideration for further display after it has been used one time. The standard functionality display_msg(msg_string(100) + msg_string(random(101, 1004)) doesn't allow us to have that kind of control. Is this the kind of task for which an array might be helpful?

Not really sure what is your task exactly (maybe your example is bad?), but yes, arrays are quite flexible.

arr := range_from_msg(NAME, 101, 104);
s := message_str(NAME, 100);
while (len_array(arr) > 0) do begin
   i := random(0, len_array(arr) - 1);
   s += arr[i];
   array_unshift(arr, i);
end
display_msg(s);

I will probably add some more library functions to use arrays as lists, stacks, etc.

 

[phobos2077]

I did a couple of bug fixes and refactoring. While at it, I also rewritten my SSL library (a bunch of utility procedures) to work with arrays. Here goes a brief documentation on library procedures and macros.

(this should be in sfall.h)

// ARRAY MACRO'S

// create persistent list
create_array_list(size)

// create temporary list
temp_array_list(size)

// create persistent map
create_array_map

// create temporary map
temp_array_map

// true if array is map, false otherwise
array_is_map(x)

// returns temp list of names of all arrays saved with save_array() in alphabetical order
list_saved_arrays

// removes array from savegame
unsave_array(x)

// true if given item exists in given array, false otherwise
is_in_array(item, array)

// true if given array exists, false otherwise
array_exists(array)

// sort array in ascending order
sort_array(array)

// sort array in descending order
sort_array_reverse(array)

From lib.arrays.h:

// push new item at the end of array, returns array
procedure array_push(variable array, variable item);

// remove last item from array and returns it's value
procedure array_pop(variable array);

// list of array keys (for lists it will return indexes 0, 1, 2, etc.)
procedure array_keys(variable array);

// list of array values (useful for maps)
procedure array_values(variable array);

// returns temp array containing a subarray starting from $index with $count elements
// negative $index means index from the end of array
// negative $count means leave this many elements from the end of array
procedure array_slice(variable array, variable index, variable count);

// remove $count elements from array starting from $index, returns $array
// rules for $index and $count are the same as in array_slice()
procedure array_cut(variable array, variable index, variable count);

// Copy a slice of one array into another (will not resize)
procedure copy_array(variable src, variable srcPos, variable dest, variable dstPos, variable size);

// create exact copy of the array as a new temp array
procedure clone_array(variable array);

// true if arrays are equal, false otherwise
procedure arrays_equal(variable arr1, variable arr2);

// returns maximum element in array
procedure array_max(variable arr);

// returns minimum element in array
procedure array_min(variable arr);

// returns sum of array elements (or concatenated string, if elements are strings)
procedure array_sum(variable arr);

// returns random value from array
procedure array_random_value(variable arr);

/**
 * Fill array (or it's part) with the same value. 
 * pos - starting position
 * count - number of items to fill (use -1 to fill to the end of the array)
 * value - value to set
 * returns arr
 */
procedure array_fill(variable arr, variable pos, variable count, variable value);

/**
 * Merge arr2 on top of the arr1
 */
procedure array_append(variable arr1, variable arr2);

/*
 Functions for working with sets (add item to set, remove from set)
 Sets are simple arrays where any value could exist only once
*/
procedure add_array_set(variable array, variable item);
procedure remove_array_set(variable array, variable item);


/**
 * Converts any array to string for debugging purposes
 */
procedure debug_array_str(variable arr);

/**
 * Saving/loading helpers
 */

// load array and create it (save) if it doesn't exist
procedure load_create_array(variable name, variable size);

// name - saved name, arr - two-dimensional array (array of arrays)
// arrays on both levels can be both lists or maps
procedure save_collection(variable name, variable arr);

// load collection previously saved with save_collection
procedure load_collection(variable name);

#define load_create_array_map(name)    (load_create_array(name, -1))

As a most useful feature for me and the main reason I started all this, here is an example of "collection" usage:

   temp := ["John", "Mike", "Kate", "Presley", "Rob", "Jim", "Steven", "Greg"];
   coll := create_array_list(0);
   for (i:=0; i<5; i++) begin
      call array_push(coll, {
         "name":  array_random_value(temp), 
         "phone": random(10000, 99999), 
         "price": (random(1000, 9999) / 100.0), 
         "something": "else"
      });
   end
   
   coll := load_collection("people");

This creates collection (you can think about it as a "list of objects" which resides in your savegame) with information about 5 people.

I will use something like this to store information about traps, but you can use this for anything like lasting visual effects (eg. smoke), some complex logic (like dynamic economy system) or anything where many of something is involved.

I can write more detailed examples on per-use case basis, if someone wants it.

To use all this in your mod, refer to this post for all required software and headers.

Additionally, I made a few tweaks in sslc compiler:
1) You can use parentheses around for and foreach loop headers (similar to most C-like languages):

for (i := 0; i < n; i++) 
    display_msg(smoke[i].color);

foreach (obj in smoke) 
    display_msg(obj.color);

These changes are backward compatible with older sfall sslc, but I advise to always use parentheses because this way code looks better and it avoids syntax error in some specific case (compiler bug).

2) Previously you couldn't break out of foreach loop, it would always iterate over all array items. Now you can using foreach - while syntax:

found := false;
foreach (obj in array while not(found)) begin
    if (...) then
        found := true;
end

So while adds additional condition before each iteration along with "index < array size" check.
This is basically a workaround for not having break/continue/goto statements in the language.

Trivia: both for and foreach are loops are basically translated into while loop with some temp variables auto-generated code at compiling. To see exactly how this works, try "roundtrip" in Sfall Script Editor (compile with sslc and then decompile with int2ssl).

And check out other new sfall 3.4 syntax features in first post.

Edit: made a few bug fixes to sslc:
1) moved unstable optimizations from Full to Experimental (it really breaks my code). It should be much safer to use Full optimization now.
2) implemented proper removal of unused imported variables.
3) namespace compression now works properly with imported/exported variables in your code.

 

[Endocore]

Good work on all this. I think your example answered my question pretty well. I have a lot of cases where I want to do something like that, randomly going through dialogue lines without wanting any particular line to be repeated until all the lines have been shown.

 

[phobos2077]

Some more thoughts about arrays after finally utilizing them in the actual mod.

Stumbled upon some pitfalls:
1. Be careful when you call fix_array, free_array, load/save_array when working with multi-dimensional arrays. Let's say we have some script scperson.ssl:

variable myArray; // this should hold "normal" array, which lives until this script is closed, but don't get saved in savegame

procedure something_p_proc begin
    ....
    // this is array expression which creates 3 temp arrays (2 maps and 1 list which holds ID's to both "maps")
    myArray := [{"something": "cool", "number": 132}, {"something": "else", "number": 3231}]; 
    // we want our array to be used later (for example, in other "_p_proc" call), so we call
    fix_array(myArray); // array is no longer temporary
end

Now if you reference myArray like: get_array(myArray[0], "something") - you expect to get "cool", but you will get zero (0) instead. Why? because there is no actual "array" data type in scripts and when you call "fix_array" you only pass ID of the top-level array (with 2 elements in it) which gets "fixed", others do not.
Solution: use foreach to iterate over top levels of multidimensional array and call fix_array on each lower-level array (I will probably add library function for this... )

The proper way to fix this would be to add new datatype to scripts, so scripting engine will know exactly when you are dealing with array and act accordingly, like in "real" languages with actual complex types support.

2. I realized that exported variables don't work and I needed a way to pass variables between scripts without actually storing them in savegame (which is not necessary for most variables). I ended up writing something like this:

variable trapv; // map

...
trapv := create_array_map;
save_array("trapvars", trapv); // save it to use in other scripts
trapv[VAR_1] := "value";
trapv[VAR_2] := 123; 

......
// in other script
trapv := load_array("trapvars");
display_msg(trapv[VAR_1]); // etc.

The problem here is that "trapvars" got saved into savegame... The proper way to do this would be to "fix" exported/imported variables with sfall, but this is just more engine hacking... An easier but dirtier solution (which I will probably end up doing) is to add some function that will make "saved" arrays not really saved (they will be available by load_array calls from any script not saved to savegames.). Still, this is an awful design and you may as well just ignore that your save will hold some unnecessary data. (it will still NOT grow larger over time like it could with old arrays).

3. "Dot" syntax is cool, but I realized that I won't use it, because it's not safe. Let me explain. Let's say we want to have a bunch of "objects" with some properties in them, you can do this:

myObject.name := "Bench";  // this actually compiles into: set_array(myObject, "name", "Bench")

But you can also make mistake when you code:

myObject.mame := "Bench"; // this will work just as well, because "mame" is just a string

With brackets syntax, I could use defines to catch such errors on compiling stage:

#define OBJ_NAME  "name"
...
myObject[OBJ_NAME] := "Bench"; // ok

// now if I make mistake
myObject[OBJ_MAME] := "Bench"; // I will receive "undeclared identifier" error right away, which is good

Latter syntax doesn't look as sexy but it is safer to use.

Programmer nonsense:

Spoiler

My idea to fix this is to add some syntax sugar to sslc compiler (not sure if it worth it) like:

structdef begin 
   variable name;
   variable hp;
end SomeObject;

// ... in code, instead of "variable":
SomeObject myObject;

myObject.mame := "Bench"; // error - undeclared property "mame" in object of type SomeObject

The problem with this is that it breaks the dynamic typing nature of Star Trek...

4. Chained array dereferencing syntax doesn't currently work:

array[1][5] := 5; // error, use set_array(get_array(array, 1), 5, 5);
array[5].name := 5; // error, use set_array(get_array(array, 5), "name", 5);
array.map[5] := 5 ; // error, use set_array(get_array(array, "map"), 5, 5);

Edit: issues #2 and #4 are no longer a problem (implemented solutions for both)