Module exports
Classes
- new DataStream()
- dataStream.map()
- dataStream.filter()
- dataStream.reduce()
- dataStream.do()
- dataStream.all()
- dataStream.race()
- dataStream.unorder()
- dataStream.into()
- dataStream.use()
- dataStream.run()
- dataStream.tap()
- dataStream.whenRead()
- dataStream.whenWrote()
- dataStream.whenEnd()
- dataStream.whenDrained()
- dataStream.whenError()
- dataStream.setOptions()
- dataStream.copy()
- dataStream.tee()
- dataStream.each()
- dataStream.while()
- dataStream.until()
- dataStream.catch()
- dataStream.raise()
- dataStream.bufferify()
- dataStream.stringify()
- dataStream.toArray()
- dataStream.toGenerator()
- dataStream.pull()
- dataStream.shift()
- dataStream.peek()
- dataStream.slice()
- dataStream.assign()
- dataStream.empty()
- dataStream.unshift()
- dataStream.endWith()
- dataStream.accumulate()
- ~~dataStream.consume()
- dataStream.reduceNow()
- dataStream.remap()
- dataStream.flatMap()
- dataStream.flatten()
- dataStream.concat()
- dataStream.join()
- dataStream.keep()
- dataStream.rewind()
- dataStream.stack()
- dataStream.distribute()
- dataStream.separateInto()
- dataStream.separate()
- dataStream.delegate()
- dataStream.rate()
- dataStream.batch()
- dataStream.timeBatch()
- dataStream.nagle()
- dataStream.window()
- dataStream.toJSONArray()
- dataStream.toJSONObject()
- dataStream.JSONStringify()
- dataStream.CSVStringify()
- dataStream.exec()
- dataStream.debug()
- dataStream.toBufferStream()
- dataStream.toStringStream()
- dataStream.toBufferStream()
- dataStream.toStringStream()
- DataStream:from()
- DataStream:pipeline()
- DataStream:fromArray()
- DataStream:fromIterator()
- new StringStream()
- stringStream.shift()
- stringStream.split()
- stringStream.match()
- stringStream.toBufferStream()
- stringStream.parse()
- stringStream.toDataStream()
- stringStream.lines()
- stringStream.JSONParse()
- stringStream.CSVParse()
- stringStream.append()
- stringStream.prepend()
- stringStream.exec()
- StringStream:SPLIT_LINE
- StringStream:fromString()
- StringStream:pipeline()
- StringStream:from()
Other symbols
- ~ShiftBufferCallback
- ~BufferParseCallback
- ~MapCallback
- ~FilterCallback
- ~ReduceCallback
- ~DoCallback
- ~IntoCallback
- ~UseCallback
- ~TeeCallback
- ~ScramjetTransformCallback
- ~ScramjetWriteCallback
- ~ScramjetReadCallback
- ~DataStreamOptions
- ~ShiftCallback
- ~AccumulateCallback
- ~ConsumeCallback
- ~RemapCallback
- ~FlatMapCallback
- ~JoinCallback
- ~AffinityCallback
- ~DelegateCallback
- ~RateOptions
- ~ExecDataOptions
- ~CreateModuleOptions
- ~StreamMixin
- ~ScramjetPlugin
- ~MultiMapCallback
- ~DistributeOptions
- ~ValueOfCallback
- ~NumberStreamOptions
- ~ShiftStringCallback
- ~ParseCallback
- ~ExecOptions
Use scramjet from scratch
This guide shows how to start using scramjet from scratch - meaning on a bare os. With this guide you'll be able to create and execute your own scramjet
pipeline on any server or computer
Step 1: Install prerequisites
Scramjet is based on node.js, so we'll need to install that before scramjet - the easiest way to do that is using nvm
so first install that:
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
source ~/.bashrc # you can restart your terminal as well.
nvm
is a cool little tool that allows you to automatically install node.js on a machine - all you need is ssh access, not even root access is needed.
There's a windows version of nvm here. It has some small differences, but I hope this guide does work with it too. If not - please report an issue
Next we need to install the current version of node, or any you like - scramjet
is compatible with all supported node versions (currently 8, 10, 12).
nvm install node # instead of node you can set this to any version you like, 8.16.0 for instance.
And now you'll have node
up and running in order to confirm that let's try:
node -v
# v12.8.0
Step 2: create a node project
There's a lot of IDE's out there - I'd recommend VSCode and GitHub's Atom. There's some who'd prefer Webstorm or Sublime and those who can actually exit vim
without restarting the whole server - in the end you will need a good text editor and who am I to judge? Choose your weapon of choice and let's now start cracking:
First let's create a directory and enter it (I'll use command line, but if you're using the IDE's browser of Nautilus
that's fine too):
mkdir ~/src/my-workflow/
cd ~/src/my-workflow/
That done we need to create a node package. This is not a necessary step, but it'll allow us to share our work with other later.
npm init # now we'll be asked a couple questions - about name etc. It's ok to just keep pressing enter.
After that you'll see a new file called package.json
that will hold out package data. Now let's install scramjet
:
npm install --save scramjet # this installs and adds scramjet to package.json so others know it's needed to run
Now after that's done we can create a simple scramjet workflow in a javascript file - for instance index.js
- for example lets download a list of files fetched from an external server:
#!/usr/bin/env node
// the first line above allows us to execute this file
const {StringStream} = require('scramjet');
const {get: get_} = require('https');
const {createWriteStream} = require('https');
// here's a simple wrapper to fetch stream from any server - you can use this or axios, request or node-fetch.
const get = (url, options = {}) => new Promise(resolve => get(url, options, resolve));
StringStream.from(async () => get('https://www.some.server.net/some.file.txt')))
.lines()
.parse(x => x.split('\t'))
.map(async ([url, data, name]) => [await get(`${url}?${data}`), name])
.each(([name]) => console.log(`downloading ${name}...`))
.each(([data, name]) => new Promise((resolve, reject) => createWriteStream(name).on('finish', resolve).on('error', reject)))
// the cool thing here is that you're writing a couple files at the same time
// see the docs (www.scramjet.org) for `maxParallel`.
.each(([name]) => console.log(`downloaded ${name}.`))
.run()
.catch(e => {
console.error(e.stack);
throw e;
});
Now let's allow to execute the file (you don't need to do that on Windows) by running chmod +x index.js
and now we can run this:
./index.js