This article will introduce you to the key principles of our data processing platform. Our data apps engine has its own approach to creation, deployment and runtime of serverless applications; in many aspects different than solutions from domains such as software buses, integration suites and Function-as-a-Service offerings. We hope you will find it interesting.

"3 in 1" data processing platform

Our core runtime environment is called "Scramjet Transform Hub". It's available as a standalone software package and will be the core element of our Scramjet Cloud Platform.

We like to call our approach "3 in 1 data processing platform" as it combines 3 concepts into one solution:

  • data processing engine
  • serverless data applications
  • complete API with CLI (covering both I/O and management endpoints)

Let's dive into each point separately.

Data processing engine

Scramjet Transform Hub creates unified deployment, runtime, management and execution plain for serverless applications (sequences).

In short, STH allows you to start data processing in 3 simple steps:

1. Deploy

si sequence send <sequence-package-tar>

2. Run

si sequence run <sequence-id>

3. Send data

curl -H "Content-Type: application/octet-stream" --data-binary "@file.txt" <instance-input-endpoint>

You are free to post to our sequence simple HTTP request, file, send a stream or even read data from other stream or API.

Please notice that, contrary to typical microservices setup, there is no expensive step of building containerized image, pushing it to registry and then downloading it to container orchestrator to run the microservice. You can move from directory with code to sequence processing your data in less than a minute.

We do package our apps but their size is measured in kilobytes, not in hundreds of megabytes as in case of container images. Light app design gives better performance, optimized resources usage and simpler CI/CD process.

We do have a short, 3 minute demo, showing the whole package preparation, deployment and run process on our YouTube channel (see Scramjet Transform Hub - Intro demo)

Serverless data applications

We call the user applications sequences. They have capabilities to perform continues data and stream processing, they have no run time limits or input data size limits.

Each sequence has a straightforward structure - it's a directory with at least two core files:

Below there is content of one of our sample "hello world" sequences, yielding integer numbers.

const { DataStream } = require("scramjet");
module.exports = async function (stream, start = 0, end = 1000) {
await DataStream.from(async function* () {
let i = +start || 0;
while (i++ < end) {
await new Promise(res => setTimeout(res, 1000));
yield { x: i };

You can find intro readme and more samples in our dedicates repository scramjetorg/scramjet-cloud-docs


Let's look at Transform Hub API via commands available in our CLI:

The above commands (and related API) covers complete management of data processing engine and serverless apps running on top of it.

Once started, each running instance exposes following API endpoints:

This approach follows "batteries included" approach, and each running instance is handled in the same way.

Why Scramjet Cloud Platform

Our approach shown above has several benefits:

As a summary, below you will find diagram showing various patters of chaining data processing on our platform: