Use Scramjet for processing forex data.

Imagine you need a stream of data from any source available on the web, be it currency prices, temperature readouts, or any other stream of data - in our example, it will be a stream of timestamps from NYC.

Let's get started. I'll show you how to scrape timestamp data, but the mechanism works for any stream from a website. Feel free to let us know what you might be using this for in the comments!

1. Install Scramjet Transform Hub and the CLI

We'll be using Scramjet CLI to pack and send the programs to the server, so go ahead and install it.

npm i -g @scramjet/cli

In order to run our sample, you'll need to install STH and start it. You can install Transform Hub on the same machine or on a remote one that you have network access to. When that's done, start it there.

# This will download and install STH npm i -g @scramjet/sth # This will start it sth # See sth --help for more info.

If you installed STH on a remote machine using the ip a command to see what IP's the machine has.

2. Get the sample from Scramjet's repo

Clone the repo locally, get into the samples directory and find scraping. We have to install our local dependencies as well.

Copy the following lines to your terminal:

# This downloads the repo to your computer git clone # This will enter the directory with the sample cd scramjet-cloud-docs/samples/crypto-prices # This will install the local dependencies npm install

3. Check out the sequence code

Now let's open up an editor to get a better understanding of what's going on inside.

# If you're viewing the file from within VSCode use this: code index.ts # Otherwise use either vi, nano or just write editor to use the default one editor index.ts

We're exporting an async generator function with the default arguments. You can think of it as just a function that returns multiple values over time. It's typed as a ReadableApp, which doesn't use the input and uses the three other parameters - the currency strings and the interval.

const app: ReadableApp<string> = async function* ( _stream, currency = "BTC", baseCurrency = "USD", interval = 3000 ) {

Note: Using the types is not required, but it makes it easier for you to write code that will work with STH.

As you can see inside, we run an infinite loop and fetch the data from the API based on the arguments. At the beginning of the loop we also set up a timer, it'll come in handy a bit later. Notice we're not awaiting it yet.

while (true) { const ref = defer(interval); const data = await fetch(`${currency}-${baseCurrency}/spot`);

Then we yield the result with a new line character appended for prettier output.

yield JSON.stringify(await data.json()) + "\r\n";

Finally, we wait for the previously set timer. Thanks to this, we'll be making requests every 3 seconds, not with a space of 3 seconds in between the requests.

4. Let's build and run.

Now when we have an idea of what's going on under the hood. Let's run our application.

We'll create a standalone package for our sequence. We can get started with building our source code or executing the typescript compiler (this step isn't necessary for plain javascript modules)

# this builds the code tsc -p tsconfig.json # this copies the package.json file to the output directory # note: the directory depends on your tsconfig.json cp -r package.json dist/ # this will install dependencies in the dist folder (cd dist && npm i --only=production)

The sample has a handy npm script that does all of the above in one simple command:

npm run build

Now, let's use SI CLI to create a package which will create a tar.gz compressed package.

si pack -o crypto-sample.tar.gz dist/

And we can now send it as a sequence like this:

# You can provide the name si seq send crypto-sample.tar.gz # we can use the "-" sign to call the last one si seq send -

Now we have an ID of our newly created sequence. Nothing runs yet but let's change that! Let's start our sequence. To do so, we need to pass ID and our arguments: URL and currency symbols. We won't pass the interval so, it will default to 3000 milliseconds as in the code.

si seq start - BTC USD

Now we can connect to the sequence output and see how it shows up:

si seq output -

In the terminal you'll see the current exchange rate every 3 seconds for every single line.

Easy peasy! In your application, you will likely be consuming our REST API. Take a look at Scramjet Transform Hub's API documentation here.

5. Let's clean up.

The program will run forever unless you stop it first. Let's try to stop the service now:

si inst stop - 3000

The last started sequence will stop in 3 seconds (that's the 3000 argument there). You can then delete the sequence afterward:

si seq delete -

Wrap up

And there you go. In this article I've shown how to write and deploy a simple data acquisition program to Scramjet Transform Hub, how to access the data.

Here's a couple of links you may want to see also:

Project co-financed by the European Union from the European Regional Development Fund under the Knowledge Education Development Program. The project is carried out as a part of the competition of the National for Research and Development: Szybka Ścieżka.


Kacper Pietrzak

Kacper Pietrzak

Backend Developer at Scramjet