tc39 / proposal-smart-pipelines Goto Github PK

See #25 and http://logs.libuv.org/tc39/2018-03-24.

Interesting idea here, with the "don't shoot yourself in the foot" reasoning. I agree that I'd rather not encourage currying as the main way to use the pipeline operator, but it seems like this syntax disallows that entirely. Is this intentional or desired?

The current context-free grammar:

Pipeline[In, Yield, Await] :
  PipelineStep[?In, ?Yield, ?Await]
  PipelineStep[?In, ?Yield, ?Await] `|>` Pipeline[?In, ?Yield, ?Await]

PipelineStep[In, Yield, Await] :
  PipelineBareFunction
  [lookahead ∉ {`{`}] ConditionalExpression[?In, ?Yield, ?Await]

…is ambiguous, despite the early error that requires PipelineStep : ConditionalExpression to contain a topic reference.

This can be fixed by changing it to:

Pipeline[In, Yield, Await] :
  CoverPipelineBareStepAndTopicStep[?In, ?Yield, ?Await]
  CoverPipelineBareStepAndTopicStep[?In, ?Yield, ?Await] `|>` Pipeline[?In, ?Yield, ?Await]

…then movingPipelineBareFunction : SimpleReference (and other bare-style productions) to a supplemental grammar:

PipelineBareStep[In, Yield, Await] :
  SimpleReference
  `new` SimpleReference [if Feature BC]
  `await` SimpleReference [if Feature BA]

PipelineBareStep[In, Yield, Await] :
  SimpleReference
  `new` SimpleReference [if Feature BC]
  `await` SimpleReference [if Feature BA]

PipelineTopicStep[In, Yield, Await] :
  [lookahead ∉ {`{`}] ConditionalExpression[?In, ?Yield, ?Await]

A static semantic rule should then, given a CoverPipelineBareStepAndTopicStep, return either a PipelineBareStep or a PipelineTopicStep.

See babel/babel#3780.

Let's go further and play with the language to parse a mapper

Christmas is coming.. 🙏

map

[1,2] ||> # + 1 |> console.log // [2,3]

const add = a => b => a + b
// bare style :)
[1,2] ||> add(1) |> console.log // [2,3]

// to
console.log([1,2].map(add(1)))

filter

[1,2] <|> (# > 1) |> console.log // [2]

const above = a => b => b > a
[1,2] <|> above(1) |> console.log // [2]

// to
console.log([1,2].filter(above(1)))

reduce

[1,2] <|| # + ## |> console.log // 3

const add = a => b => a + b
[1,2] <|| add |> console.log // 3

// bare bare bare
// [1,2] <|| add(#)(##) |> console.log // 3

// to
console.log([1,2].reduce((a,b) => a + b))

From IRC #tc39 today with @littledan:

littledan (Daniel Ehrenberg): …I think this could partly be addressed by a simpler-looking document layout (e.g., maybe split off the explainer pieces about follow-on proposals into separate md documents?) …

jschoi (J. S. Choi): Regarding the explainer: I can try separating the additional features into their own explainer documents. This is probably long overdue, but I have been focusing on the specification for the past week. I’ll make an issue for this in the smart-pipelines repository.

jschoi (J. S. Choi): The specification is currently a single document too, with an annex for each additional feature. Is this similarly too confusing or overwhelming for the reader? Should I consider separating the specification also? Perhaps I should.

littledan (Daniel Ehrenberg): I think the specification being in one document vs multiple documents is less important, as many fewer people read and understand the specification compared to the explainer. The specification is usually unintelligible to most audiences, and difficult to the rest of them, no matter how you cut it.

jschoi (J. S. Choi): Noted; thank you.

littledan (Daniel Ehrenberg): oh, another contradiction: explaining everything fully vs concisely emphasizing the important stuff

jschoi (J. S. Choi): The real-world examples hopefully help concretize how it would look, but…yes.

10:35 AM littledan (Daniel Ehrenberg): the readme is very long! this means you have spoken to many things, but a portion of the audience will just be unaware of your thought process. This is an inherent difficulty

10:36 AM jschoi (J. S. Choi): That’s true too. Maybe I should make an explainer explainer, pfft. Or separate the examples…? I think the examples do help make it seem more compelling.

Yes. Hopefully the readme split will help.

At least on a communicative level, though not so much at a fundamental-tradeoff level.

An advantage that this proposal has is that parentheses can never change the meaning of a code without giving an early error. x |> (await #) is the same as x |> await #, and x |> (await y) and x |> await y are both early errors that require the developer to disambiguate between x |> (await y)(#) and x |> await y(#). This should be noted in Goals and in the Motivation, Core Proposal too, as well as the pipeline-operator wiki.

Edit: That advantage is moot. However, this proposal still has an advantage in that it forces the writer to make explicit which of these reasonable alternatives do they mean by x |> foo: x |> foo(#, y), x |> foo(y, #), and x |> foo(y)(#). This is also important, so I’m reusing this issue to track better explaining this in § Goals, in § Motivation, Core Proposal, and in the wiki. See #2 (comment).

@js-choi at least could you allow the syntaxe with a plugin option ? it's handy and the # give a parsing error in prettier so we can't format our code, it should be a prettier issue but as this # is not 'definitive' nor pure functionnal we would have to remove it anyway when using F# pipelines or this one in pure bare style

To leave a particular example, how would this interoperate with stateful primitive-like operators like distinct or scan? I know it's not directly related to this proposal, but if we go with my lifted pipeline strawman as an extension of this, how would it interop with them? (The current syntax offers no convenient escape hatch to break the "smart" default, instead requiring a new name to be assigned.)

// Usage
books >:> distinct() >:> console.log(#)

// Implementation
function distinct(by = Object.is) {
    let hasPrev = false, prev
    return x => {
        const memo = hasPrev
        hasPrev = true
        return memo || by(prev, prev = x) ? [x] : []
    }
}

The current draft’s Feature PF uses +> to create unary-parameter functions. This means that arguments after the first argument are discarded. This may result in surprising behavior, especially given that +> is intended to address method extraction. For instance, (+> console.log)(5, 2) is equivalent to ($ => $ |> console.log)(5, 2), which prints only 5. This is not equivalent to console.log.bind(console), contrary to what the explainer currently says. In addition, this makes Feature PF without Feature NP incompatible with adding Feature NP in the future.

The spec needs to be changed so that Feature PF creates variadic functions that apply all their arguments to the first. +> console.log would instead be equivalent to (...$) => console.log(...$), and (+> console.log)(5, 2) would correctly print 5 and 2.

Much of the work is already done, because even the Core Proposal allows for multiple topics. Feature PP would also need to be changed such that, if a pipeline’s lexical environment has multiple topics, then all of them are applied to the pipeline’s body.

I'm struggling to see the reasoning for feature PP. Adding it lets people save exactly one character whenever it's used, but introduces ASI hazards and, imo, makes things look slightly more confusing in general. (I find myself looking for the topic in the examples that use it, then realizing that there is no topic and inferring that it's using PP.)

If you're not paying close attention to things, you can accidentally miss that something is a nested prefix pipeline - x |> foo + ( |> bar(#) ) can, depending on formatting, be mistaken for x |> foo |> bar(#).

If I'm not mistaken, this also allows topic-form to not include a #: x |> (|> bar) would use the outer topic, but wouldn't express it literally. Catch-22 here: if this is valid, it breaks the expectation that topic-form always has a # in it; if it's not valid, it's a confusing instance where prefix-pipeline can't be used due to surrounding context. (That is, x |> # + (|> bar) would be valid, so the error depends on surrounding context.)

It also complicates the variadic-handling of pipeline functions, as you note in #4 (comment).

I also think it ends up being somewhat confusing with pipeline-functions; both of them start with a pipeline-ish operator, but one creates a function to obtain the topic, the other just immediately uses the surrounding topic. In other words, I see xs.map(|> foo) being a footgun with PP, where the author meant xs.map(+> foo). Without PP this is a syntax error as written; if correctly written as xs.map(# |> foo), it's much more obvious what's happening.

(I can't find the link right now, but at least one person in the pipeline-proposal repo was asking for |> foo to automatically create a pipeline function, so the confusion is already possible.)

Overall, I just don't see the particular value of being able to omit the # from the beginning of a pipeline, considering the downsides.

See #1.

Consider bringing back right associativity so that x |> (f |> g) is valid. x |> f |> g would still also be valid. This may make any future short-circuiting conditional pipeline operator like with x ?> f |> g—as well as monadic binding, Kliesli arrows, or other sorts of higher-order composition—easier to explain. Right associativity was first introduced in 2254707 and removed in 90ada79.

@littledan has asked whether there is any way to reconcile the F#-style-only proposal edited by @mAAdhaTTah with this F#-plus-Hack-style “smart-mix” proposal and make the latter a superset of the former. It seems likely that presenting the pipe operator to TC39 incrementally might be more likely to earn consensus for approval. That is, it might be desirable to present the simpler F#-style-only pipe proposal first, then, assuming TC39 accepts it, later also proposing a Hack-style smart-mix pipe proposal as an extension.

The obvious way to combine F# style with Hack style into a “smart mix” is to rely on the presence of the topic operator to determine which style a pipe uses. But I am still quite afraid of this from a human usability perspective; it’s a mode error waiting to happen. I suspect that it would be easy for human developers to miss the presence or absence of the topic operator, requiring careful scanning of the entire pipe expression to ensure its meaning, and undermining syntactic locality and semantic clarity.

That is why the current smart-pipe proposal restricts its tacit F# mode to identifiers. However, this choice breaks compatibility with @mAAdhaTTah’s F#-only proposal.

Another common concern people have expressed about the current smart-pipe proposal is that it disallows tacit use of metafunctions. An example of such concern may be found at tc39/proposal-pipeline-operator#134 (comment). Under the current smart-pipe proposal, using autocurried metafunctions would require explicit use of the topic reference. For instance, given divideBy, powerOf, and multiplyBy (unary functions that create unary functions), and assuming the current smart-pipe proposal using % for the topic, a developer would have to write code like this:

100|>divideBy(2)(%)|>powerOf(2)(%)|>multiplyBy(-1)(%)

It might be true that there are general issues with autocurrying in variadic-functional languages. And the idiomaticity of autocurrying in the APIs of JavaScript, the DOM, Node, etc. might be questionable. But it’s still very much worth figuring out whether the autocurrying use case can be made easier.

Most importantly, it is worth carefully thinking about whether backward compatibility can be achieved with an F#-only proposal, while avoiding human-usability hazards.

The answer is that yes: There is at least one alternative way to reconcile the two piping styles with one common operator: by marking placeholder expressions with brace-delimited blocks. Object literals can’t ever be functions, so this would be visually unambiguous…at least less so than relying on the presence of %.

In this case, |> would have tighter precedence than binary (and perhaps prefix/postfix) operators—basically functioning as a slightly looser member access—such that these are equivalent:

x|>o.m + 1

(x|>o.m) + 1

Phase One

In the initial proposal, tacit function application would look like this (assuming F# style):

x|>o.m

o.m(x)

x|>o.m(a)

o.m(a)(x)

x|>o.m(a)()

o.m(a)()(x)

x|>await

await x

x|>% + 1

SyntaxError: Unexpected %

x|>{o.m}

SyntaxError: Unexpected {

x|>{o.m(%, a)}

SyntaxError: Unexpected {

x|>{% + 1}

SyntaxError: Unexpected {

With this initial proposal, the example in tc39/proposal-pipeline-operator#134 (comment) could use the tacit syntax, because it doesn’t use braces:

100|>divideBy(2)|>powerOf(2)|>multiplyBy(-1)

Phase Two

A follow-up proposal for placeholders would look like this:

x|>o.m

o.m(x)

x|>o.m(a)

o.m(a)(x)

x|>o.m(a)()

o.m(a)()(x)

x|>await

await x

x|>% + 1

SyntaxError: Topic % is
used in pipe expression
without braces; surround
pipe expression with
braces to bind topic

x|>{o.m}

SyntaxError: Pipe
expression binds topic
% but does not use
topic; pipe expressions
surrounded by braces
always bind topic

x|>{o.m(%, a)}

o.m(x, a)

x|>{% + 1}

x + 1

Like with the current smart-pipe proposal, braces with placeholders would allow Elixir-style first-argument function calls, but they would be explicit, not tacit. The Elixir-style example from this comment by @zenparsing would look like:

import { map, filter, collect } from 'iter-funs';
someIterator|>{map(%, cb1)}|>{filter(%, cb2)}|>collect|>{%.forEach(cb3)};

Phase Three

If we eventually get to this point, I hope that tacit pipe functions, like those in Additional Feature PF, could eventually be considered by TC39 too. They would be compatible with using braces for placeholders, with just one additional operator:

+>o.m

(...$rest) => o.m(...$rest)

+>o.m(a)

(...$rest) => o.m(a)(...$rest)

+>o.m(a)()

(...$rest) => o.m(a)()(...$rest)

+>{o.m(%, a)}

o.m(x, a)

+>{% + 1}

$ => $ + 1

a.map(+>{%.toLowerCase()})

a.map($ => $.toLowerCase())

+>% + 1

SyntaxError: Topic % is
used in pipe expression
without braces; surround
pipe expression with
braces to bind topic

+>{o.m}

SyntaxError: Pipe
expression binds topic
% but does not use
topic; pipe expressions
surrounded by braces
always bind topic

Phase Four

And, assuming that Phase Three is adopted, then N-ary pipes (Additional Feature NP) would make them even more useful.

a.sort(+>{% - %%})

a.sort(($, $$) => $ - $$)

a.sort(+>{%.localeCompare(%%)})

a.sort(($, $$) => $.localeCompare($$))

const debug =
  +>{console.log('[debug]', ...)};
debug(1, 2, 3);

const debug =
  (...$rest) =>
    console.log('[debug]', ...$rest);
debug(1, 2, 3);

Questions

Some problems with braces would be that:

• Braces would make it more difficult to create object literals, which would require nested grouping operators like with x|>{{a: %, b}}. But this would not be unique to pipes. Arrow functions already do something similar with x => ({a: x, b}).

• Braces may make developers think that they could add statements into the braces, since they look like regular blocks. Developers might expect x|>{ console.log(%); % + 1 } to just work, but it would be a syntax error. This might occasionally be annoying but at least the error is an early error. And if do expressions ever get accepted then placeholder pipes could be extended to support statement lists with do-like semantics.

Other questions include:

• How tight should the precedence be? Consider the following examples, which assume that the operator’s tightness is between the binary and unary operators:

  100|>divideBy(2)|>powerOf(2)|>multiplyBy(-1)
  
  someIterator|>{map(%, cb1)}|>{filter(%, cb2)}|>collect|>{%.forEach(cb3)}
  
  !flag|>processFlag
  
  !state.done || !(num0|>greaterThan(num1))
  
  num|>Math.log|>{new Message(%)}
  
  iterator|>{map(%, cb1)}|>{%.forEach(callback)}

  If the operator is tightened to between unary operators and method access, then the examples become:

  100|>divideBy(2)|>powerOf(2)|>multiplyBy(-1)
  
  someIterator|>{map(%, cb1)}|>{filter(%, cb2)}|>collect|>{%.forEach(cb3)}
  
  (!flag)|>processFlag
  
  !state.done || !num0|>greaterThan(num1)
  
  num|>Math.log|>{new Message(%)}
  
  iterator|>{map(%, cb1)}|>{%.forEach(callback)}

  If the operator is further tightened to have the same precedence as member access, then they become:

  100|>divideBy(2)|>powerOf(2)|>multiplyBy(-1)
  
  someIterator|>{map(%, cb1)}|>{filter(%, cb2)}|>collect|>{%.forEach(cb3)}
  
  (!flag)|>processFlag
  
  !state.done || !num0|>greaterThan(num1)
  
  num|>(Math.log)|>{new Message(%)}
  
  iterator|>{map(%, cb1)}.forEach(callback)

  Which tradeoffs would generally be best?

• Would -> be a better choice for the operator than |>? Other programming languages use -> as a very tight operator for member access or related concepts, such as with C++’s pointer->MemberFunction(). If a tight precedence is chosen for the pipe operator, especially if it's equally or nearly as tight as method access, then -> might be better than |> at conveying the analogy between member access and function/expression application.

  100->divideBy(2)->powerOf(2)->multiplyBy(-1)
  
  someIterator->{map(%, cb1)}->{filter(%, cb2)}->collect->{%.forEach(cb3)}
  
  (!flag)->processFlag
  
  !state.done || !num0->greaterThan(num1)
  
  num->(Math.log)->{new Message(%)}
  
  iterator->{map(%, cb1)}.forEach(callback)

• Should the tacit syntax favor F# style (as im the examples above) or Elixir style? It is probably a fundamental trade off that both tacit Elixir style and tacit F# style cannot be equally accommodated by the same operator. The example in tc39/proposal-pipeline-operator#134 (comment) would be the following if the tacit syntax favored F# style:

  100|>divideBy(2)|>powerOf(2)|>multiplyBy(-1)

  …but would change to the following if the syntax favored Elixir style:

  100|>divideBy(2)()|>powerOf(2)()|>multiplyBy(-1)()

  And tc39/proposal-pipeline-operator#143 (comment) would be the following if the tacit syntax favored F# style:

  import { map, filter, collect } from 'iter-funs';
  someIterator|>{map(%, cb1)}|>{filter(%, cb2)}|>collect|>{%.forEach(cb3)};

  …and if it favored Elixir style:

  import { map, filter, collect } from 'iter-funs';
  someIterator|>map(cb1)|>filter(cb2)|>collect|>{%.forEach(cb3)};

I don’t know what I’d call this brace-using idea, but I think I might already like it more than the current smart mix proposal, even if it’s slightly chunkier.

And it would automatically be a superset of @mAAdhaTTah’s current F#-only proposal (as long as the proposal tweaks its precedence and adds two early errors for cases that would never naturally occur in F# style alone anyway).

Fill in the blank TODOs in term rewriting.

"Unfortunately, these deeply nested expressions often result in messy spaghetti code, due to their mixing of prefix, infix, and postfix expressions together. Writing such code requires many nested levels of indentation and parentheses. Reading such code requires checking both the left and right of each subexpression to understand its data flow."

This statement is an exaggerated view of the situation and IMHO driven by coding style rather than using simpler coding techniques that are defined in and driven by the proposal itself. For example, the following nested code, similar to the example given:

let testDataset = [ 'abc\n', 'def\n', 'ghi\n' ] ;
async function * toLine ( iterable ) { /* implementation */ }
async function * toUpperCase ( iterable ) { /* implementation */ }
async function toLogger ( iterable ) { /* implementation */ }
await toLogger(
  toUpperCase(
    toLine(
      testDataset
    )
  )
) ;

can simply be re-written as:

let testDataset = [ 'abc\n', 'def\n', 'ghi\n' ] ;
async function * toLine ( iterable ) { /* implementation */ }
async function * toUpperCase ( iterable ) { /* implementation */ }
async function toLogger ( iterable ) { /* implementation */ }
let temp = testDataset ;
temp = toLine( temp ) ;
temp = toUpperCase( temp ) ;
temp = toLogger( temp ) ;
await temp ;

This coding style has zero indentation, thus negating the overstatement in the proposal.

The general purpose of this proposal is to transform the result of the left expression and make it a parameter of the next function in the sequence, which is what the simpler coding style is doing:

let testDataset = [ 'abc\n', 'def\n', 'ghi\n' ] ;
async function * toLine ( iterable ) { /* implementation */ }
async function * toUpperCase ( iterable ) { /* implementation */ }
async function toLogger ( iterable ) { /* implementation */ }
await ( 
  testDataset
  |> toLine
  |> toUpperCase
  |> toLogger
) ;

I put parenthesis around the proposed syntax because it is unclear what using await in this situation means, e.g., is that await testDataset (no effect) or await the whole pipeline?

See tc39/proposal-pipeline-operator#110 (comment), tc39/proposal-pipeline-operator#107 (comment), tc39/proposal-pipeline-operator#101 (comment).

I had forgotten that a discussion about robust method pre-extraction happened last September. By the time they returned to it in November, time was up. January was skipped while preparing for this, though @ljharb gave a comment about the importance of the prefix ::, as well as recently that it must be robust against delete Function.prototype.bind. ljharb/function.prototype.name gives examples of this pattern. There has been discussion of method-binding caching in tc39/proposal-bind-operator#46, in which a WeakMap is associated with each realm.

In any case, pipelines, including smart pipelines, and robust cached method pre-extraction can coexist. Using ljharb/function.prototype.name/implementation.js, line 19 as an example of useful coexistence:

return fn
|> &Function.prototype.toString
|> &String.prototype.match(#, classRegex)
|> !!#;

…which is just:

return !!&String.prototype.match(
  &Function.prototype.toString(fn),
  classRegex);

This security robustness is quite a different use case than simply wanting to express the callbacks in promise.then($ => console.log($)) more tersely, as well as the other goals of smart pipelines as well as the goals of other pipelines: most importantly, the untangled composition of expressions and calls.

I think secure cached method binding is out of scope for smart pipelines, even with Feature PF. It is also out of scope for @mAAdhaTTah’s Proposal-1 pipelines. These are orthogonal use cases. They cannot kill each other.

See https://tc39.github.io/ecma262/#sec-algorithm-conventions-syntax-directed-operations.

Feature BP conflicts with Core Proposal for pipelines whose bodies are object literals, like this: x |> { a: # }.

Arrow-function ConciseBody solves this by requiring object literals to be parenthesized. It’s probably reasonable for us to do the same. Object literals as pipeline bodies would be much less common than blocks.

The spec and explainer need to be updated in their Core Proposal sections.

the readme contains 2 links labeled 'do expressions' (a,b) under Relations to Other Works, both point to ./relations.md#do-expressions, in which there are again two links which point to the paragrapf itself

EDIT: turns out there actually are two seperat blocks about do expressions. they are pretty much redundant

The do expressions proposal is at Stage 1, and there are some serious concerns that some TC39 members have about it (especially about completion values). Does this proposal depend on do expressions? I don't see do expressions as part of the specification here. If there is no dependency, I'd suggest moving examples based on do expressions into a separate section, to make the lack of dependency clear.

Q1
Can we use/is there a reason why we can't use this as the topic reference keyword instead of #? I think that the current use of this in JavaScript and the topic reference here are analogous. In addition, consider that this proposal would in practice cover many of the same use cases as the bind operator proposal, which uses this in the spot analogous to where # is used here.

Q2
Why was # chosen in particular?

I'm surprised by the inclusion of this construct. It seems a bit complicated to me, and it ends up necessitating disallowing arbitrary expressions on the right hand side of a pipeline (#2). I can see that it's used in some of your examples, but I wonder if it would be OK to just include an explicit (#) for these.

See tc39/proposal-pipeline-operator#101 (comment).

See tc39/proposal-pipeline-operator#144.

Some examples use ??., which may confuse some people who are not up to date on optional chaining. Smart pipelines are not dependent on optional chaining (just like how they’re not dependent on do expressions, cf. #1). The examples need to be rewritten to not use ??..