Thoughts on Scramjet 5.0

The time has come to start development of the next major version of the Scramjet framework. 4.x versions have brought stability and clear vision of the range of applications.

Support

Scramjet v4 will be supported with security updates provided until April 2019.
Most of the software written against v4 should work flawlessly in v5.
A -compat module may be released if sufficient interest shows (it'll be v4 interface using v5 underneath).
Deprecated usages will result in immediate error if possible.

Changes

With the upcoming v5 we expect the following changes:

Input/output streams

One of the issues in using current version is that usage of DataStream.pipeline or a number of transforms returns just the last of a chain of streams instead of a full stream.

In scramjet v5 it will be possible to write new chunks into the result of a stream, like this:


_14const stream = new DataStream()
_14    .map(x => x + 1)
_14    .each(console.log)
_14    .run();
_14
_14stream.write(1);
_14stream.write(2);
_14stream.write(3);
_14stream.end(4);
_14
_14// 2
_14// 3
_14// 4
_14// 5

In most of the operations a stream that will be returned will be a Duplex stream that will allow writing data before any transforms and will allow reading data already transformed.

Better transform merge

Multiple transforms are being merged in a single stream object. In v5, as long as the stream just gets piped to another stream that is able to receive an array of methods it will get that. Slicing of that array will happen at runtime so that you'll be able to pipe any point in the stream flow.

Strict options checking

Streams will no longer accept any options, only those defined will be accepted.

Any access (read or write) to an undeclared option will result in error
Options will be declared per class and will inherit along the classes' inheritance chain

Drop support for class override

Some methods, especially map allowed passing of a Clazz argument. This looks fairly nice on the face of it, but it does make the method quite unpredictable. This is also a feature that is inconsistently implemented in the framework.

Arguments destructuring

New options destructure_args, restructure_args and structure_args will be provided to allow more complex operations to be simplified on sub-objects.

Extraction of extra streams

Usage of the WindowStream introduced in 4.14 series did not pick up too much, similarly NumberStream will be removed.

These streams will still be possible for use with the help of scramjet plugins as those will be introduced as separate modules.

Plugin refactor

Currently plugins inject their methods into prototypes of core streams. This causes that no two conflicting plugins may run in the same process. The refactor will address that issue.

Source generation

Every method for each stream will be placed into a separate file - the resulting class will be constructed by

Core fat-loss diet

Core currently contains way too much methods. In scramjet-core v5.0.0 all main streams will be removed and only the PromiseTransformStream will be exposed.

All main methods will get moved to the main scramjet module.

Stack property

All streams will now expose a stack property showing place of construction.

Array compatibility

Array.prototype inspired methods like map/reduce/find etc. will now follow the ECMA interface. For example the MapCallback will now receive index and self arguments providing index in stream and the stream being mapped respectively.

Similarly the map method will accept thisObj argument, as well as any additional arguments that will be passed to the MapCallback.

Consistent stream graphing

Currently there's the referrer option and graph methods available. These work one way, so you can see the object that constructed the prior object, but finding where the stream is targeted (what other streams are consuming it) is rather hard.

Method graph will change it's operation returning a soup of all stream inter-connections - graph object will be shared between all streams.

sources and targets methods will be introduced (names to be clarified).

Project co-financed by the European Union from the European Regional Development Fund under the Knowledge Education Development Program. The project is carried out as a part of the competition of the National for Research and Development: Szybka Ścieżka.