608B71FC-006A-4934-A643-7D9BA9340450Blog

Scramjet Transform Hub Release 0.17

blog__author-img
Michał Czapracki
CEO at Scramjet, Data Streaming Expert.
23F2E8CD-3026-46A5-86CC-D13114F7176E425AE875-B1A1-4EA1-8529-075D08DA0BB1

22 February 2022

This is a small but important release that allows brings the Python runner nearly up to full functionality and an improvement to the client that allows calling Scramjet Transform Hub directly from the browser!

Python runner update

The Python runner now fully supports input and output data, stdio is well handled and well tested.

The new runner now supports:

  • Graceful stop - running si inst stop on a python instance no longer kills the sequence unconditionally
  • Health monitoring - now you can expose data for si inst health from Python!
  • Event handling - si inst event and similar operations will now allow event handling and messaging
  • Dependencies handling - now sequences can use proper PYTHONPATH dependency resolution.

So drop your index.py in your favorite flavor of vi and paste this:


_10
from scramjet.streams import Stream
_10
_10
def run(context, input):
_10
return Stream.read_from(input).map(lambda s: f"Hello {s}!")

We need to allow Python to work without Scramjet Framework, but this will come in an upcoming release.

Browser client

Previously the Scramjet API Client would work only from node.js. The main reason for this was using node-fetch as one of the dependencies. We do believe it's a fantastic module and one that should be included in the base Node API long time ago. Since we do want to support node.js 14 and 16 at least until the end of life we needed to find an alternative solution. We managed to find @sapphire-fetch - a simple way to support a proper W3C Fetch API without the hassle of having to deploy multiple modules.

Minor changes

  • Fixed identification failure when STH is started with existing sequences
  • New automated tests for Python runner have been added
  • API Docs improvements

Kubernetes!

I'm happy to announce that STH will gain a really important feature: you will be able to run stream processors on any Kubernetes cluster. We're working on proper documentation of how to start that up, but currently you will have access to the three execution adapters:

  • docker - the default for running sequences in a standard server space
  • process - the adapter running non-separated processes for simple deployments
  • kubernetes - for deployment in cloud environments and leverage multiple servers

The kubernetes adapter is expected to come early next week, so keep checking. :)

Photo by jggrzProject co-financed by the European Union from the European Regional Development Fund under the Knowledge Education Development Program. The project is carried out as a part of the competition of the National for Research and Development: Szybka Ścieżka.

Related Posts