Transform Hub, welcome to the Scramjet Family!

Scramjet Transform Hub (STH) is designed to drastically improve and simplify what has been possible with the Scramjet Framework, giving developers even more tools and freedom in the Cloud. Unlike the current Serverless systems, Scramjet Transform Hub gives you an opportunity to process data from different sources and in multiple scenarios. You can not only separately process small events, but also lists of local files, large files, pull streams from streaming APIs and so on. The choice is yours and STH adapts to your use case natively.

We looked at the competition and compared their solutions to our newly launched STH. This is a lot to process, but most importantly STH offers best-in-class performance metrics at an unrivalled cost while being simple to deploy, in just 4 steps! If Scramjet Transform Hub sounds interesting to you, go and check it out – we believe you’ll be able to unlock great new use cases with STH.

See how to get started on the Scramjet GitHub Repository.

Feature comparison across major STH competitors

Examples Short explanation Competition product and its type Scramjet
Type - FaaS Open-source FaaS Serverless Streams Transform as a Service
Product - Lambda, Azure Fn, GCF OpenFaaS, Nuclio Kafka, Kinesis, Azure EH Transform Hub
Unlimited data size Can process data in chunks, so there’s no data size limit at all. Data is simply processed while it’s being uploaded to the platform. no no no yes
Sequential processing Allows you to set up sequences of consecutive functions and reuse processing plugins/modules when you perform repetitive computations or accessing similar data sources. no no no yes
Concurrent processing Doesn’t force you to run a separate process for every single data point. You can process multiple items simultaneously and use backpressure to control the concurrency level. no no no yes
Built-in backpressure Automatically pauses and resumes data sources and uploads to match the processing time. If you’re bottlenecked by an API limit or a slow database, the process will halt previous operations and uploads until the load goes back to normal. no no no yes
Long running processes/ Statefull operation Thanks to the fact that processes are not limited to a single data point and can run for long time (days or even weeks), you make use of stateful processing without having to rely on external systems introducing massive latencies and complexity. no no yes yes
Direct public IP operation Allows you to expose IP ports on external IPs. You can therefore mimic any existing protocol, even if it’s UDP based. no no no yes
Programmable data acquisition Doesn’t rely on external data integration. The data acquisition is fully integrated with the platform which means you can develop your own data integrations, either by polling a database or reading external URLs or any other source you can reach in a programmatic way. no no no yes
Bring your own machine Is available as installable software that can run on any machine you provide, but also, through adapters, you can use it to deploy programs on a swarm of devices of your choice. Thanks to the open-source nature of STH the applications are endless. no yes maybe yes

Performance metrics comparison across major STH competitors

Mertic Lambda, Azure Fn, GCF OpenFaaS, Nuclio Kafka, Kinesis, Azure EH Scramjet Transform Hub
Data size limit 6 MB RAM 1 MB Unlimited
Inter-step processing latency (95th) 100 ms 100 ms N/A ~ 10 μs
Maximum processing time 15 min 1 h N/A Unlimited
End-to-end latency level (w/o net). How much library (frameworks, caches, buffers) memory is needed to sustain operations of an 8 step workflow processing 8 concurrent items (compared to 1x). 64 x 64 x 8 x 1 x
8 x 8 step external orchestrators. How many additional running systems are needed to sustain operations of an 8 step workflow processing 8 concurrent items. 9 x 9 x 9 x 1 x
8 x 8 step serialization operations. How many times the data needs to be serialized and deserialized in an 8 by 8 scenario. 128 x 128 x 128 x 16 x

Pricing model comparison across major STH competitors

Metric Lambda, Azure Fn, GCF OpenFaaS, Nuclio Kafka, Kinesis, Azure EH Scramjet Transform Hub
Unit of scale Event Data Mixed N/A
Timescale seconds seconds hrs / days N/A
Pricing model complexity V-Complex Open Source Complex Open Source
Scenario low throughput monthly: 0.5 requests per second + 500 ms processing + 512 MB RAM + 10 kb output on average $1,11 Servers cost = $ 300 N/A Free + servers cost = $100
Scenario generating data for ML 10 GB of items: 500 ms processing + 512 MB RAM + 100 kb output on average $1,34 N/A
Scenario large data output: 0.5 requests per second + 1000 ms processing + 512 MB RAM + 1 MB output on average $113,82 N/A
Scenario full month of ops: 1 requests per second + 1000 ms processing + 512 MB RAM + 100 kb output on average $26,57 N/A
Scenario 50% baseline + 8x scale: Like full month but with 8x peak for 4 hours a day $60,08 N/A
Scenario 1 TB of i/o ASAP: Best case for processing 1TB of input and 1TB of output data (with additional services when needed) $154,13 $159,65
Project co-financed by the European Union from the European Regional Development Fund under the Knowledge Education Development Program. The project is carried out as a part of the competition of the National for Research and Development: Szybka Ścieżka.

Author:

Budleigh Salterton

Budleigh Salterton

The founder and inventor of Scramjet