Use scramjet from scratch

This guide shows how to start using scramjet from scratch - meaning on a bare os. With this guide you'll be able to create and execute your own scramjet pipeline on any server or computer

Step 1: Install prerequisites

Scramjet is based on node.js, so we'll need to install that before scramjet - the easiest way to do that is using nvm so first install that:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
source ~/.bashrc # you can restart your terminal as well.

nvm is a cool little tool that allows you to automatically install node.js on a machine - all you need is ssh access, not even root access is needed.

There's a windows version of nvm here. It has some small differences, but I hope this guide does work with it too. If not - please report an issue

Next we need to install the current version of node, or any you like - scramjet is compatible with all supported node versions (currently 8, 10, 12).

nvm install node # instead of node you can set this to any version you like, 8.16.0 for instance.

And now you'll have node up and running in order to confirm that let's try:

node -v
# v12.8.0

Step 2: create a node project

There's a lot of IDE's out there - I'd recommend VSCode and GitHub's Atom. There's some who'd prefer Webstorm or Sublime and those who can actually exit vim without restarting the whole server - in the end you will need a good text editor and who am I to judge? Choose your weapon of choice and let's now start cracking:

First let's create a directory and enter it (I'll use command line, but if you're using the IDE's browser of Nautilus that's fine too):

mkdir ~/src/my-workflow/
cd ~/src/my-workflow/

That done we need to create a node package. This is not a necessary step, but it'll allow us to share our work with other later.

npm init # now we'll be asked a couple questions - about name etc. It's ok to just keep pressing enter.

After that you'll see a new file called package.json that will hold out package data. Now let's install scramjet:

npm install --save scramjet # this installs and adds scramjet to package.json so others know it's needed to run

Now after that's done we can create a simple scramjet workflow in a javascript file - for instance index.js - for example lets download a list of files fetched from an external server:

#!/usr/bin/env node
// the first line above allows us to execute this file

const {StringStream} = require('scramjet');
const {get: get_} = require('https');
const {createWriteStream} = require('https');

// here's a simple wrapper to fetch stream from any server - you can use this or axios, request or node-fetch.
const get = (url, options = {}) => new Promise(resolve => get(url, options, resolve));

StringStream.from(async () => get('https://www.some.server.net/some.file.txt')))
    .lines()
    .parse(x => x.split('\t'))
    .map(async ([url, data, name]) => [await get(`${url}?${data}`), name])
    .each(([name]) => console.log(`downloading ${name}...`))
    .each(([data, name]) => new Promise((resolve, reject) => createWriteStream(name).on('finish', resolve).on('error', reject)))
        // the cool thing here is that you're writing a couple files at the same time
        // see the docs (www.scramjet.org) for `maxParallel`.
    .each(([name]) => console.log(`downloaded ${name}.`))
    .run()
    .catch(e => {
        console.error(e.stack);
        throw e;
    });

Now let's allow to execute the file (you don't need to do that on Windows) by running chmod +x index.js and now we can run this:

./index.js