The Fastest Route to Hub API: Using the Hub Client in Application Context

The Scramjet Transform Hub (STH) empowers developers to deploy and manage powerful data processing programs. A key tool in this process is the Scramjet SDK. Scramjet SDK allows to easily control your Sequences and Instances, but also in you are using Scramjet Cloud Platform, also your Space and Hubs. This provides powerful ways to implement code execution orchestration and monitoring.

Many existing serverless solutions provide SDK, but Scramjet goes one step further and provides also Scramjet SDK in your application context! This lets you send requests to your STH and Space API internally, without through the proxies, public API gateway or even API server reducing data transfer speed significantly.

Scramjet SDK available in the application context has two instances: Scramjet Hub Client and Scramjet Space Client.

Scramjet Hub Client lets developers send requests internally to STH API. It can be used to start your Sequences, terminate Instances, collect logs and generally perform all STH operations that are exposed through STH API, but much faster and without the need to configure Scramjet SKD Hub Client. This way to you can all possible API requests but only to the STH where the Instance that uses Scramjet Hub Client is deployed.

Another SDK available in the application context - Space Client is even more powerful. It lets developer to control all Space where the Instance that uses it is deployed. From the convenience of your code, without the need to configure Scramjet SDK, you can not only list all Hubs connected to your Space, manage and monitor resources, but also control every Hub in your Space! So you can start Sequences, terminate Instances, and so on, on any Hub that is connected to your Space. You can for example from the cloud supervisor Instance, deploy and start your program on a Hub located on a different cloud, or on the edge environment.

In this blog post we will cover Hub Client, since its mechanisms or working are similar to Space Client but simpler. It will give us a good ground for learning about the powerful Space Client in the next blog post.

A Closer Look at the Implementation

In our example, a JavaScript Sequence is deployed on STH to initiate another Sequence within the same STH. The asynchronous function, which is the main function of this Sequence, aims to start another Sequence identified by sequenceId, a parameter passed to it. First the Sequence Client is retrieved using the provided sequenceId by utilizing the method this.hub.getSequenceClient. Then, the Sequence start method is triggered, effectively launching its execution. If successful, the function uses the this.logger method to log a confirmation message. In case of errors, it logs the error message and provides detailed error information through instance log endpoint.


_29const { PassThrough } = require("stream");
_29
_29module.exports = async function(_input, sequenceId) {
_29    this.logger.info("Working...");
_29    const out = new PassThrough();
_29
_29    try {
_29        const seqClient = await this.hub.getSequenceClient(sequenceId);
_29
_29        this.logger.info("Sequence client called:", seqClient);
_29        const seqStart = await seqClient.start(sequenceId);
_29        const seqStartStr = JSON.stringify(seqStart);
_29
_29        this.logger.info(`Sequence ${sequenceId} started: ${seqStartStr}`);
_29        
_29        const instClient = await this.hub.instClient(retInstObj._id);
_29        const instHealth = await instClient.health();
_29        const instinfo = await instClient.getInfo(); 
_29        
_29        this.logger.info(`Instance health: ${instHealth}, Instance info: ${instinfo}`);
_29        
_29        out.write(seqStart._id); // write instance id to output stream
_29        out.end();
_29    } catch (e) {
_29        // if error occurs it will be logged under /instance/:id/log endpoint (CLI: si inst log <instID>)
_29        this.logger.error(e);
_29    }
_29    return out;
_29};

Explanation:

1. Asynchronous Function: The function takes input and sequenceId as input and initiates the sequence based on the provided Sequence-id.
2. Retrieving Sequence Client: The hub.getSequenceClient method fetches the client object associated with the given Sequence-id.
3. Sequence Initiation: The seqClient.start method starts the desired Sequence.
4. Handling Results: Upon successful initiation, the Sequence-id is written to the stream and the stream is closed, indicating completion.
5. Error Handling: Catches and logs any errors encountered during the process.

Conclusion

In conclusion, Scramjet SDK offers the flexibility to create a powerful supervisory Sequence that not only monitor other Sequences but also possess the ability to restart them upon failure. This functionality, combined with the capability to initiate and manage other Sequences, makes such a Sequence ideal for automating long-running tasks like data synchronization and scheduled jobs. Furthermore, Scramjet's long-lived Instances ensure the supervisor Sequence remains operational throughout the process, providing a reliable and robust solution for managing complex workflows.

Project co-financed by the European Union from the European Regional Development Fund under the Knowledge Education Development Program. The project is carried out as a part of the competition of the National for Research and Development: Szybka Ścieżka.