Use Scramjet for processing forex data.

Imagine you need a stream of data from any source available on the web, be it currency prices, temperature readouts, or any other stream of data - in our example, it will be a stream of timestamps from NYC.

Let's get started. I'll show you how to scrape timestamp data, but the mechanism works for any stream from a website. Feel free to let us know what you might be using this for in the comments!

1. Install Scramjet Transform Hub and the CLI

We'll be using Scramjet CLI to pack and send the programs to the server, so go ahead and install it.

In order to run our sample, you'll need to install STH and start it. You can install Transform Hub on the same machine or on a remote one that you have network access to. When that's done, start it there.

If you installed STH on a remote machine using the ip a command to see what IP's the machine has.

2. Get the sample from Scramjet's repo

Clone the repo locally, get into the samples directory and find scraping. We have to install our local dependencies as well.

Copy the following lines to your terminal:

3. Check out the sequence code

Now let's open up an editor to get a better understanding of what's going on inside.

We're exporting an async generator function with the default arguments. You can think of it as just a function that returns multiple values over time. It's typed as a ReadableApp, which doesn't use the input and uses the three other parameters - the currency strings and the interval.

Note: Using the types is not required, but it makes it easier for you to write code that will work with STH.

As you can see inside, we run an infinite loop and fetch the data from the API based on the arguments. At the beginning of the loop we also set up a timer, it'll come in handy a bit later. Notice we're not awaiting it yet.

Then we yield the result with a new line character appended for prettier output.

Finally, we wait for the previously set timer. Thanks to this, we'll be making requests every 3 seconds, not with a space of 3 seconds in between the requests.

4. Let's build and run.

Now when we have an idea of what's going on under the hood. Let's run our application.

We'll create a standalone package for our sequence. We can get started with building our source code or executing the typescript compiler (this step isn't necessary for plain javascript modules)

The sample has a handy npm script that does all of the above in one simple command:

Now, let's use SI CLI to create a package which will create a tar.gz compressed package.

And we can now send it as a sequence like this:

Now we have an ID of our newly created sequence. Nothing runs yet but let's change that! Let's start our sequence. To do so, we need to pass ID and our arguments: URL and currency symbols. We won't pass the interval so, it will default to 3000 milliseconds as in the code.

Now we can connect to the sequence output and see how it shows up:

In the terminal you'll see the current exchange rate every 3 seconds for every single line.

Easy peasy! In your application, you will likely be consuming our REST API. Take a look at Scramjet Transform Hub's API documentation here.

5. Let's clean up.

The program will run forever unless you stop it first. Let's try to stop the service now:

The last started sequence will stop in 3 seconds (that's the 3000 argument there). You can then delete the sequence afterward:

Wrap up

And there you go. In this article I've shown how to write and deploy a simple data acquisition program to Scramjet Transform Hub, how to access the data.

Here's a couple of links you may want to see also:

• A YouTube video showing the above example
• More data acquisition and stream processing samples to review
• The readme and initial documentation to start with
• The Scramjet Transform Hub landing page
• Wanna help? Here's some stuff we'd be happy if you helped us