Scramjet Transform Hub 0.15

We just released a new version of Scramjet Transform Hub. The new version fixes a couple of important bugs and is aimed at a better developer experience.

CLI with last argument memory

The current version of CLI allows usage of - sign to specify that the command refers to the last item added. For instance when packing a new sequence, you can just upload it without specifying the file path.

A couple of cases you may want to try:

• si pack /path/to/your/program && si seq send - - this will automatically send the packed sequence to the server
• si seq send ./file.tar.gz && si seq start - - this in turn will start the last sequence sent
• si seq start $LAST_ID && si seq logs - - this will get logs from the last sequence executed by the user.

The si command stores the last id and allows you to reuse it in subsequent command by providing - instead of the id the same way as cd - command works. We hope this will massively improve the CLI experience, as there's no longer need to parse the previous command line if we want to do quick deployment of a sequence.

Python runner WIP

Python is coming to STH. We decided to release a very early version of the Python runner and currently, the options are quite limited. In order to run STH sequences written in python, you need to create a python file that exposes a def run function that executes the stream processor you want to run. The API is 1:1 the same as in JavaScript albeit some parts are not yet fully implemented.

We hope this will open the floodgates for our friends in the DataOps and MLOps communities and we're happy to get in touch with early beta testers. We're working on a new Discord server and we'd like to hear from you there as soon as we open it so please keep checking our website, LinkedIn etc.

Bugfixes

• Fix STH to be able to run from within a docker container (bugged since 0.13 and introduction of TCP Runner). Now STH should run in docker quite well, just remember to run it with docker sock as mentioned in the docs.
• STH is able to reconnect to Scramjet Cloud Platform automatically. The Platform will allow you to connect multiple STH's together, even if they are behind a firewall. We're getting ever closer to releasing it to the public and keep checking our website for updates - we're going to open a beta not long from now.

Previous version 0.14

We intend to keep posting about the new features of Scramjet Cloud Platform, just to clarify what went in when 0.14 version was deployed.

New features:

• Improved typedoc definitions and documentation - thanks to this writing sequences will be much more intuitive in Visual Studio or other IDE's that support TypeScript.
• Removal of BPMux - in order to improve efficiency of running instances, connected streams (including data, stdio and process control) are no longer multiplexed.
• Verser module - a simple server that allows exposing your local http server without listening on a public IP address. The server connect to the client, so the roles are reversed, hence the name verser.
• Published package tests - now we have the possibility to run our BDD tests against public packages in npm. This will allow us to confirm that the public STH is not regressive bugged and hopefully not bugged at all, coverage permitting...

Bugfixes and minor improvements:

• STH: Server will now listen on 0.0.0.0 instead of localhost by default.
• Runner: Fixed data input and output encoding (broken in 0.13.3)
• CLI: fixed possibility to stream input data to stdin (broken since 0.13) - previously filename was required.
• Host: Semi gracefully stop API servers on Host stop
• General: Massive tsdoc documentation effort done across most important packages.
• Workflow: added local docker builds for pre-runner and runner images
• Code style: removed some minor rules that were annoying.
• Runner: Fix ending stdin

Not yet there...

Yes, as always there are things that fall behind the schedule. This week we worked hard but we didn't manage to get the Object Logger. Since we wrote STH every component logger just plain text messages. Now we plan that logs will no longer be conveyed as text, but will be sent as JSON strings for better filtering, coloring, etc. in the future. The new format will be brought to you with the next release, but we'll make sure that CLI will still output the same format as previously.

During next two weeks we'll be focusing on major stability fixes and finalizing the python runner. If you'd like to test, reach out to us on discord or GitHub.