Proposal for lifted pipelines
.map
/.filter
/etc....map
/.then
-like stuff, using coll :> func
+ @@lift
..merge
/.combine
-like stuff, using Object.combine(...colls, func)
+ @@combine
..filter
/.takeWhile
/.flatten
, using coll >:> func
+ @@chain
.coll :> async func
/Object.asyncCombine
/coll >:> async func
, with matching @@async{Lift,Combine,Chain}
symbols for each.coll :> await func
↔ await (coll :> async func)
, coll >:> await func
↔ await (coll >:> async func)
.The proposal is in three parts:
coll :> func
, which calls coll[Symbol.lift](func)
.@@lift
to control the above..map
..then
.Object.combine(...colls, func)
which delegates to coll[Symbol.combine](other, func)
@@combine
to control the above._.zip
.Promise.join
.Observable.combineLatest
coll >:> func
, which calls coll[Symbol.chain](func)
@@chain
to control the above..flatMap
crossed with .filter
and Lodash's _.takeWhile
..then
..mergeAll()
crossed with .filter
and .takeWhile
.Async variants exist for each:
coll :> async func
calls Symbol.asyncLift
instead of Symbol.lift
.Object.asyncCombine
calls Symbol.asyncCombine
instead of Symbol.combine
.coll >:> async func
calls Symbol.asyncChain
instead of Symbol.chain
.The two operator variants can use await
instead of async
in async
functions as sugar:
coll :> await func
↔ await (coll :> async func)
.coll >:> await func
↔ await (coll >:> async func)
.Original es-discuss thread (previously, this was specific to function composition, but I've since generalized it.)
Before I continue, if you came here wondering what the heck this is, or what the point of it is, I invite you to read this blog post about composition and this one on monads, and I encourage you to google both concepts. Long story short, yes, it's a thing, and yes, it's pretty useful for a variety of reasons.
Also, note that this is meant to work as a complementary extension of the existing pipeline operator proposal. I use the F# variant here for simplicity, but there are still multiple competing syntaxes.
Function composition has been used for years, even in JS applications. It's one thing people have been continually reinventing as well. Many utility belts I've found have this function in particular, most common ones have it:
_.compose
_.flow
and _.flowRight
R.compose
and R.pipe
There's also the numerous npm modules and manual implementations (it's trivial to write a basic implementation). Conceptually, it's pretty basic:
function pipe(f, g) {
return (...xs) => g(f(...xs))
}
It lets you do turn code like this:
function toSlug(input) {
return encodeURIComponent(
input.split(" ")
.map(str => str.toLowerCase())
.join("-")
)
}
to this:
const toSlug = [
_ => _.split(" "),
_ => _.map(str => str.toLowerCase()),
_ => _.join("-"),
encodeURIComponent,
].reduce(pipe)
Or, using this proposal:
const toSlug =
(_ => _.split(" "))
:> ^.map(str => str.toLowerCase())
:> ^.join("-")
:> encodeURIComponent(^)
Another scenario is when you just want to define a long stream:
// RxJS Observables
import {Observable} from "rxjs"
function randInt(range) {
return Math.random() * range | 0
}
function getSuggestions(selector) {
const refreshElem = document.querySelector(".refresh")
const baseElem = document.querySelector(selector)
const refreshClickStream = Observable.fromEvent(refreshElem, "click")
const responseStream = refreshClickStream.startWith()
.map(() => `https://api.github.com/users?since=${randInt(500)}`)
.flatMap(url => Observable.fromPromise(
window.fetch(url).then(response => response.json())
))
.map(() => listUsers[randInt(listUsers.length)])
return Observable.fromEvent(baseElem, "click")
.startWith(undefined)
.combineLatest(responseStream, (_, listUsers) => listUsers)
.merge(refreshClickStream.map(() => undefined).startWith(undefined))
.map(suggestion => ({selector, suggestion}))
}
Observable.of(".close1", ".close2", ".close3")
.flatMap(selector => getSuggestions(selector))
.forEach(({selector, suggestion}) => {
if (suggestion == null) {
// hide the selector's suggestion DOM element
} else {
// show the selector's suggestion DOM element and render the data
}
})
Problem is, there's this massive boilerplate, complexity, and jQuery-like tendencies inherent with nearly every reactive library out there. RxJS has attempted to compromise with a .do(func)
/.let(func)
that's the moral equivalent of a |>
operator, but even then, using custom operators doesn't feel as natural as built-in ones. (jQuery and Underscore/Lodash have similar issues here, especially jQuery.) Using this proposal (all three parts) + the pipeline operator proposal + the observable proposal, this could turn out a bit easier and lighter:
// This proposal (35 SLoC)
function randInt(range) {
return Math.random() * range | 0
}
function eachEvent(elem, event) {
return new Observable(observer => {
const listener = e => observer.next(e)
elem.addEventListener(event, listener, false)
observer.next()
return () => elem.removeEventListener(event, listener, false)
})
}
const refreshElem = document.querySelector(".refresh")
const refreshClickStream = fromEvent(refreshElem, "click")
const randomItem = list => list[randInt(list.length)]
const listUsers = (async () => refreshClickStream
:> `https://api.github.com/users?since=${randInt(500)}`
:> await window.fetch(^)
:> await ^.json()
:> randomItem(^)
)()
Observable.of([".close1", ".close2", ".close3"])
:> selector => document.querySelector(selector)
>:> (async elem => Object.combine(
refreshClickStream :> {elem}
Object.combine(
fromEvent(baseElem, "click"), await listUsers,
(_, suggestion) => ({elem, suggestion})
)
}))
|> ^.subscribe({elem, suggestion}) => {
if (suggestion != null) {
// show the selector's suggestion DOM element and render the data
} else {
// hide the selector's suggestion DOM element
}
})
For comparison, here's the RxJS and callback equivalents:
// RxJS (29 SLoC + dependency)
import {Observable} from "rxjs"
function randInt(range) {
return Math.random() * range | 0
}
function getSuggestions(selector) {
const refreshElem = document.querySelector(".refresh")
const baseElem = document.querySelector(selector)
const refreshClickStream = Rx.Observable.fromEvent(refreshElem, "click")
const responseStream = refreshClickStream.startWith()
.map(() => `https://api.github.com/users?since=${randInt(500)}`)
.flatMap(url => Rx.Observable.fromPromise(
window.fetch(url).then(response => response.json())
))
.map(listUsers => listUsers[randInt(listUsers.length)])
return Rx.Observable.fromEvent(baseElem, "click")
.startWith(undefined)
.combineLatest(responseStream, (_, listUsers) => listUsers)
.merge(refreshClickStream.map(() => undefined).startWith(undefined))
.map(suggestion => ({selector, suggestion}))
}
Rx.Observable.of(".close1", ".close2", ".close3")
.flatMap(selector => getSuggestions(selector))
.subscribe(({selector, suggestion}) => {
if (suggestion == null) {
// hide the selector's suggestion DOM element
} else {
// show the selector's suggestion DOM element and render the data
}
})
// Callbacks (31 SLoC)
function randInt(range) {
return Math.random() * range | 0
}
const refresh = document.querySelector(".refresh")
function getSuggestions(selector, send) {
const elem = document.querySelector(selector)
send(elem, undefined)
async function getUsers() {
const url = `https://api.github.com/users?since=${randInt(500)}`
const response = await window.fetch(url)
const listUsers = await response.json()
return listUsers[randInt(listUsers.length)]
}
let current = await getUsers()
send(elem, current)
refresh.addEventListener("click", () {
current = undefined
send(elem, undefined)
send(elem, current = await getUsers())
}, false)
elem.addEventListener("click", () => send(elem, current), false)
}
for (const selector of [".close1", ".close2", ".close3"]) {
getSuggestions(selector, (elem, suggestion) => {
if (suggestion != null) {
// show the first suggestion DOM element and render the data
} else {
// hide the first suggestion DOM element
}
})
}
It's not much longer than the RxJS variant, but critically, it has zero dependencies beyond polyfills
So, we've got several ways of transforming values within things:
list.map(x => f(x))
- Transform the entries of an array.promise.then(x => f(x))
- Transform the value of a promise.observable.map(x => f(x))
- Transform the values emitted from an observable.stream.pipe(map(x => f(x)))
- Transform the values in a Node stream (where map
is through2-map
).func.compose(x => f(x))
- Transform the return value of a function (where .compose
is a theoretical Function.prototype.compose
).If you squint hard enough, they are all variations of this same theme: object.transform(x => f(x))
. What does that bring us?
My proposal for this is to add a new syntax with an associated symbol:
// What you write:
x :> f
// What it does:
function pipe(x, f) {
if (typeof func !== "function") throw new TypeError()
return x[Symbol.lift](x => f(x))
}
It doesn't look like much, but it's incredibly useful and freeing with the right method implementations.
name
s out of an array of records? Use array :> ^.name
.stream :> ^.target.value
.contents
of? Use func :> ^.contents
.value
and make it take events instead? Use (e => e.target.value) :> setValue(^)
.Set
of numbers and strings, and you only want numbers? Use set :> Number(^)
If you want to dig deeper into what this really does and what all it entails, this contains more details on the proposal itself.
Sometimes, you might have a couple collections, promises, or whatever things you have that hold data, and you want to combine them. You want to join them. This .combineLatest
looks like your sweet spot. Or maybe Bluebird's Promise.join
is that missing piece you were looking for. Or maybe, you just wanted to run through a couple lists without pulling your hair out. That's what this is for. It takes all those nice and helpful things, and lifts them up to where the language understands it itself. Fewer nested loops, easier awaiting, and easier zipping iterables (which is harder than it looks to do correctly).
When you squint hard enough, these start to run together, and it's why I have this:
_.zipWith(array, other, (a, b) => ...)
- Lodash's _.zipWith
observable.zip(other, (a, b) => ...)
- RxJS's _.zip
Observable.combineLatest(observable, other, (a, b) => ...)
- RxJS's Observable.combineLatest
Promise.join(a, b, (a, b) => ...)
- Bluebird's Promise.join
My proposal is to add a couple new builtins with related symbols:
// Combines each of `...args` using their related `Symbol.combine` method
Object.combine(...args, (...values) => ...)
// Combines each of `...args` using their related `Symbol.asyncCombine` method,
// returning a promise resolved with the return value
Object.asyncCombine(...args, (...values) => ...)
These are pretty straightforward, and their comments explain the gist of what they do. If you want more details about this proposal, or just want to read a little deeper into what the implementation might look like, take a look here.
Of course, mapping and combining things is nice, but they're weak sauce. They do nothing to go "no more", and they offer no facility to go "nope, not passing that along". They also don't let you go "hey, add this into the mix, too". .map
isn't enough; you want more. You want to not simply combine, but also flatten, but also filter. That's where this comes in.
After doing a bit of research to see what they really build off of, I managed to narrow it down to a single operation. Here's how I formulated that into a proposal:
// What you write:
x >:> f(^)
// What this does (roughly):
function chain(x, f) {
if (typeof func !== "function") throw new TypeError()
return x[Symbol.chain](value => {
const result = f(value)
// break
if (result == null) return undefined
// emit values (optimization)
if (Array.isArray(result)) return result
// flatten value
if (typeof result[Symbol.chain] === "function") return result
throw new TypeError("invalid value")
})
}
It's not as simple and foolproof to implement as the first two, but here's how you use it:
null
/undefined
.This helper makes it possible to filter, flatten, and truncate things generically. For example, the common .takeWhile
you find for collections and observables could be generically translated into a very simple helper:
// Use like so: `coll >:> takeWhile(^, cond)`
function takeWhile(cond) {
return x => cond(x) ? [x] : undefined
}
This isn't the only one, there's several other helpers that become trivial to write, which may change how you find yourself manipulating collections in some cases.
Also, there is an async variant that awaits both the result and its callbacks before resolving, coming in two flavors: x >:> async func
(returns promise) and x >:> await func
(for async
/await
, awaits result). This variant is itself non-trivial, not because the basic common functionality is complex, but due to various edge cases, and it's the only non-trivial facet of this entire proposal.
.map
/.filter
/etc... ()It's much more general and open. I wanted to seek the lowest denominator for working with collections, one that keeps the user's cognitive overhead at a minimum and one that didn't make the implementation of various operators difficult. Yeah, it's nice to have high-level operators as methods, but I wanted to provide the means for people to define what looping is like for their types, so they can be looped over similarly to iterables and friends, with all the same set of control flow possibilities. Specifically:
I wanted to keep it flexible and meaningful semantically.
undefined
value.)I wanted to decouple the object from the action, and more so, the type from the action.
for
loops and the like. You don't need to make assumptions about the object you have apart from that one clear interface. (If they have a weird .forEach
or a non-traditional .filter
, you don't need to care about that.)I wanted to pick the minimal pragmatic solution that solved the problem as a whole.
Array.prototype.filter
's basic functionality with this proposal, that's a good thing.)Of course, partial userland solutions have existed for a while for several of these issues with compatibility and extensibility (for observables + variant, many basic data structures, thenables, iterables + async variant), but this is an attempt to unify most of these under a single umbrella in a way that feels like JS, something that just fits right in without being too out there and unusual (especially in light of other recent proposals). Furthermore, even though it is possible to implement this in userland, it's not ideal, hence the need for a new standard:
Most in-language implementations of function composition involve a .reduce
or equivalent out of necessity since they are almost always variadic. In this scenario, engines commonly end up seeing the value as megamorphic, whether in for
or the native .reduce
, because there's only two independent IC feedback points, and because of this, it ends up hitting the slow path every single time. A native assist would be invaluable for this.
Engines have had so much trouble with optimizing Array builtins in the past, and userland implementations are even slower than that. With this proposal, the intermediate values are inaccessible unless the symbols are overridden, making optimization opportunities easier.
.map
and .filter
while eliding the intermediate function allocation and has-property check, because they can't just navely do a for loop - they also have to skip missing properties. And given the fact loop bodies can delete elements mid-loop, they had to first provide the ability to bail out from within existing optimized code into a slow path corresponding that same segment of optimized code, which is easier said than done. (No other engine does this IIUC.)Userland standards tend to be much better at working us into this ugly problem. We need fewer of those.
Symbol.iterator
, leads me to write the same exact methods about 5 times over for repeated variants.{done, value}
, really?)I know it's a common criticism that function composition, and even this proposal as a whole, doesn't need new syntax. There are in fact tradeoffs involved. But here's why I elected to go with syntax:
This is meant to mirror the pipeline operator in appearance. It's not an exact one-to-one correspondence, but I specifically want to encourage people to view it as not dissimilar to a pipeline.
There's fewer parentheses and tokens in general involved, especially if the operator is lower-precedence. Instead of a pair of parentheses for each chain + commas for each call, it's a single infix token. Also, there's fewer cases of nested parentheses, something that tends to plague functional JS.
There's less to polyfill, since you only need a statically analyzable runtime helper. The binary nature of the operators make it possible to not also have to account for a variadic application. (The Object.combine
and Object.asyncCombine
implementations are good examples of why this is the case.)
Operators are in their nature less verbose than functions, and in general, this proposal aims to keep things simple without getting too verbose. It also tries to keep from becoming unreadable, and line noise is something I wish to avoid. (In fact, the proposal tries to avoid being too tacit, requiring you to be explicit what you do at each step.)
And of course, there are downfalls to using syntax to express this:
It's not possible to use with polyfills alone. This alone will draw people away from this proposal, because they either strongly resent transpiling in general, or they just don't want to have to add yet another Babel plugin just to use it. Trust me, I get the pain, too. I've written entire 50K+ SLoC projects solo targeting ES5, and I've have historically had very little need for the ES6+ syntax additions. (About the only things I've really found myself wanting are generators, arrow functions, and async
/await
.) And yes, transpilers are usually a pain to set up, especially Babel and TypeScript and especially with existing projects with complex build systems.
The operator looks a bit foreign and/or weird. I'm fully aware the operator looks pretty arcane if you're just looking at it without added context, and doesn't obviously imply any sort of substantially modified pipeline. I'm not wanting to design a Haskell or Perl extension, so if you have better ideas, please tell me. I really want to hear it.
>=>
is too close to Haskell's Kleisli composition operator visually, which is equivalent to composing Symbol.chain
callbacks to make a new such callback. (I initially proposed this in the mailing list, and it confused functional people.)>:>
was initially used for the base pipeline operator here, but it started to look a little too Perl-like, and was a little too verbose to merit being "better" than a simple utility function.->>
was once proposed, but the reverse conflicts with unary negation (think: f <<- g
vs f << -g
). It also doesn't visually imply piping, even though it does imply a direction.>>
and >>>
may seem incredibly obvious, but you can't use them without potentially breaking a lot of existing code (they are the bit-wise arithmetic and logical right shifts, respectively). |
also falls in a similar wheelhouse as the bitwise exclusive or operator (it also happens to be a major asm.js dependency).It adds to an already-complicated language, and can be fully implemented in userland without the assistance of operators. Most things that are userland-implementable don't really need to become language constructs, and very few things would actually benefit from being a core extension.
These are just ideas; none of them really have to make it.
Object.box(value)
()An Object.box(value)
to provide as an escape hatch which also facilitates optional propagation through the various operators. Here's an example with help from the optional chaining proposal:
// Old
function getUserBanner(banners, user) {
if (user && user.accountDetails && user.accountDetails.address) {
return banners[user.accountDetails.address.province];
}
}
// New
function getUserBanner(banners, user) {
return Object.box(user?.accountDetails?.address?.province)
:> banners[^]
}
This gets uncomfortably verbose...
null
values are censored to undefined
for consistency.
undefined
to null
, so there's nothing to account for there.The returned object's prototype would implement the following:
.value
: get the underlying valueSymbol.iterator
: yield the underlying value if not undefined
, then returnSymbol.lift
:
undefined
, return this
.Symbol.asyncLift
:
undefined
, return this
.Symbol.combine
:
undefined
, return this
.Symbol.asyncCombine
:
undefined
, return Promise.resolve(this)
.Symbol.chain
:
undefined
, return this
.Symbol.asyncChain
:
undefined
, return Promise.resolve(this)
.Engines could with type feedback elide the entire pipeline and generate optimal assembly (think: zero-cost abstraction in optimized code) with little effort provided ICs assert Object.box
and the relevant symbols aren't touched.
Depending on whether cancellation turns out to include sugar syntax, this could hook into and integrate with that, adding an extra optional argument to all symbol hooks (like Symbol.lift
, etc.) to allow handling cancellation (if they support it). This could allow much better cleanup in the face of cancellation, like closing sockets or aborting long polling loops.
fantasy-land/map
method, although it's a little more permissive.async
/await
in LiveScript: https://gist.github.com/isiahmeadows/0ea14936a1680065a3a3
This is most certainly not on its own little island - even the introduction shows this. Here's several other existing proposals that could potentially benefit, or in some cases, be truly amplified, from this proposal, whether via being able to integrate with this well to its benefit, enhancing and complementing this proposal itself, or just being generally useful alongside it:
Function pipelining:
this
binding/pipelining: https://github.com/tc39/proposal-bind-operator
this
-based pipelines, I'll likely transition to built-in method helpers instead of operators to fit it better visually. (I'd go with x::lift(f)
/x::chain(f)
/x::asyncChain(f)
where const {lift, chain, asyncChain} = Object
- they're easy enough to destructure out.)Pattern matching: https://github.com/tc39/proposal-pattern-matching
Observables: https://github.com/tc39/proposal-observable
Cancellation: https://github.com/tc39/proposal-cancellation
Extra collection methods: https://github.com/tc39/proposal-collection-methods