Scramjet v4.20
The new release includes one new method unorder and some new features for pull and use.
As always there was a couple dependencies updated, including papaparse used by scramjet for CSV parsing.
One of the new methods available on all streams is unorder. It allows transforms to be processed without taking output order into account (although they are still started as a single item). The reason for the new method came from a request from one of the users in scramjet issue #25.
After some discussion we decided to add a new method to support the case in which someone would like to process a number of chunks in parallel, but is not concerned about the order in which the items are exported. For example let's consider a parallel download scenario:
StringStream.from(fs.createReadStream("downloads.txt"))
.lines()
.parse(x => x.split("|"))
.setOptions({maxParallel: 3})
.unorder([url, filename] => {
const out = request(url).pipe(fs.createWriteStream(filename));
return new Promise(out);
})
.run();
When the above code is executed scramjet will keep 3 parallel operations, but will allow a long running operation not to lock the stream. If we used map instead of unorder chunk 4 would wait for ending of operation 1, but with unorder if only one of the previous operations concludes, another can be started using a limit.
For more info see the repo with an example here: github:devinivy/scramjet-experiments.
Another change in the newer version of scramjet is allowing the use of ES6 generators in pull and use. Now it's possible to write a code like this:
// here we get a stream of files
DataStream.from(gulp.src("path")).into(async (out, x) => {
// let's use a scramjet module that read a file to generator
return out.pull(async function* () {
// first we can push some synchronous items
yield `filename=${x}`;
// then use async generators (node v10+)
const gen = await readFileToGenerator(x);
yield* gen();
yield `--end--`;
});
}, new DataStream());
It's a cool feature that we found very useful in a case for one of our customers where a workaround had to be used. Now it's fully implemented and supported in both use and pull.
