Streams in TADS 3

Using my TADS 3 Streaming Module, you can use streams for storing data. I also have an example program demonstrating some features.

The performance is excellent - in a project, I've been able to stream rougly 500 objects (and plenty of associated data) to and from a FileStream with no noticable delay.

Using streams

Streams work a bit like files, in that they allow reading and writing of data. However, they work at a higher level, providing methods for storing and loading many kinds of Tads values, including generic objects.

Furthermore, streams provide a generic interface that can be used for storing data in many different ways.

This stream module provides a FileStream class, a ByteArrayStream class and a VectorStream class, for streaming to and from byte arrays, vectors and raw files.

Each of the specific classes only have to take care of reading and writing simple integers; the Stream class takes care of the rest. (The subclasses included in the stream module also provide their own methods for storing and loading strings, but that's only for effeciency - subclasses are not required to do so.)

Value registration

Since the subclasses of Stream only knows how to work with simple integers, the Stream class must have a way of converting generic Tads values to integers.

Some types convert easily, such as true, nil, integers and strings. Some types can be decomposed into simpler values, namely lists and some objects.

But enums, properties, functions, and most objects can't simply be split into components or saved as integers. To handle these values, the Stream class provides a registration mechanism.

For instance, if you have some enums that you want to store, you'd do something like this:

enum Foo, Bar, FooBar;
...
Stream.registerValues(1000, Foo, Bar, FooBar);
...
myStream.storeValue(FooBar);

Values are registred globally; that is, once registred, they're available for all streams. Hence registerValues is called on the Stream class, rather than on an instance.

For more information, see registerValues.

Streaming objects

First of all: You can only stream generic data-only objects, and registred objects. But registred objects are considered constant. If you put a registred object on a stream, only its ID is stored, not its properties.

To store generic data-only objects, you must register any superclasses that contain code, all properties containing data, and the TadsObject class.

Here's an example, where an instance of MyClass is stored on a stream. The instance can be stored, because it only contains data, and because the superclass (MyClass) and the data-property (&foo) is registred.

class MyClass: object
    foo = nil
    print = "Foo: < foo >>
"
;

...

Stream.registerValues(5000, TadsObject, MyClass, &foo);
local stream = new VectorStream(new Vector(32, 32));
local obj = new MyClass();
obj.foo = 42;
stream.storeValue(obj);

The Stream class

streamCharset - Holds the CharacterSet used by some Stream subclasses for streaming strings. By default, all streams use UTF-8.

registerValues(id, [vals]) - Register values for later storage/retrieval. id specifies the integer ID that should be associated with the first value, the following values will be associated with consecutive IDs.

Stream.registerValues(5000, TadsObject, Vector, MyClass);
will associate TadsObject with ID #5000, Vector with #5001 and MyClass with #5002.

The call to Stream.registerValues will often be placed in the beginning of the main() function, or in a PreinitObject's execute() method.

storeList(lst) - Stores a list to the stream. A list can be streamed if all its elements can be streamed.

loadList() - Loads a list from the stream.

storeString(str) - Stores a string to the stream.

loadString() - Loads a string from the stream.

storeRegistred(val) - Stores a value registred using registerValues to the stream.

loadRegistred() - Loads a value registred using registerValues from the stream.

storeValue(val) - Stores any kind of value to the stream.

loadValue() - Loads any kind of value from the stream.

cannotStoreValue(val) - This is called when the stream can't store a value (because it's not registred, and can't be decomposed into streamable parts.)

By default, cannotStoreValue throws an EStreamStoreUnregistredValue exception.

cannotLoadType(type) - This is called when the stream doesn't recognize a value type. This often means that the underlying data have been corrupted.

By default, cannotLoadType throws an EStreamError exception.

Stream primitives

The stream primitves must be implemented by the Stream subclass employed. That is, unlike the higher level load/store methods, the following methods don't have default implementations. If the stream doesn't support a method, it'll throw an EStreamUnsupportedMethod exception.

storeByte(x) - Store a byte (8 bits).

storeWord(x) - Store a word (16 bits/2 bytes).

storeInteger(x) - Store a full Tads integer (32 bits/4 bytes).

loadByte() - Load a byte (8 bits).

loadWord() - Load a word (16 bits/2 bytes).

loadInteger() - Load a full Tads integer (32 bits/4 bytes).

getSize() - Returns the size of the stream. This will often be in bytes, but doesn't have to. Passing getSize to setPosition will make the next store/load operation occur at the end of the stream.

getPosition() - Returns the current position in the stream. The return value can later be passed to setPosition.

setPosition(pos) - Changes the current stream position.

closeStream() - This method frees any resources associated with the stream, such as file handles. By default, it does nothing.


The ByteArrayStream class

The ByteArrayStream class provides streaming to and from an underlying ByteArray. The constructor takes a ByteArray object.

myStream = new ByteArrayStream(new ByteArray(1024));

ByteArrayStream supports all stream primitives.

The VectorStream class

The VectorStream class provides streaming to and from an underlying Vector. The constructor takes a Vector object - note that the stream can't expand the Vector as neccessary, so you must specify an initial length. The VectorStream is not terribly usefull, except for debugging purposes, where you want to see what exactly ends up on the stream.

myStream = new VectorStream(new Vector(64, 64));

VectorStream supports all stream primitives.

The FileStream class

The FileStream class provides streaming to and from a raw file, or streaming from raw resource. The constructor takes a File object, which must be opened in raw mode using File.openRawFile() or File.openRawResource(). The FileStream is where streaming becomes really usefull, since you can store objects in a file and load them into a later session, or even load objects from one T3 program into another.

myStream = new FileStream(File.openRawFile('mydata.dat', FileAccessWrite));

FileStream supports all stream primitives.


The EStreamError exception

The EStreamError is the general-purpose basic Stream exception from which the other Stream exceptions inherit.

The EStreamLoadUnregistredValue exception

The EStreamLoadUnregistredValue exception is thrown on attempts to load values from a stream that have not been registred.
This indicates a corrupted stream or that different value-registrations were used when storing the value.

The EStreamStoreUnregistredValue exception

EStreamStoreUnregistredValue is thrown on attempts to store values to a stream that can only be stored when registred.
You must ensure that the value itself is registred, or that it can be decomposed into registred values.

The EStreamUnsupportedMethod exception

This exception is thrown if you try to call a stream primitive not supported by the stream. The stream classes in the streams module support all the stream primitives, so you should not see this error unless you use other stream classes.