Dynamic forking: inconsistent behavior

[pavel]

Environment

Highland: 3.0.0.beta-5
Browser: Chrome 61

Example

const numbers = Stream();
const even = numbers.fork().filter(x => x % 2 === 0);
const odd = numbers.fork().filter(x => x % 2 !== 0);

even.each((x) => { console.log("even", x); });
odd.fork().each((x) => { console.log("odd", x); });

numbers.write(1);
numbers.write(2);

console.log("> five is added");
const five = odd.fork().filter(x => x === 5);

console.log("> late write");
numbers.write(4);
numbers.write(3);

console.log("> late write 2");
numbers.write(6);
numbers.write(5);

console.log("> consume five");
five.each((x) => { console.log("five", x); })

The output:

odd 1
even 2
> five is added
> late write
even 4
> late write 2
> consume five
odd 3
even 6
odd 5
five 5

Problem

During "late write" step even path is getting processed, while on "late write 2" step it is paused.
This is inconsistent as it should be either paused from the beginning or getting processed all the time (which in my opinion is an option that grants a broader set of capabilities).

[vqvu]

Yeah, late forks have always had this issue. What's going on here is that the late fork doesn't exert back-pressure immediately since it comes in late; the decision to generate a new value has already been made. That is, before > five is added, both even and odd had already requested a value from number, so the new five couldn't stop it.

I can see why this behavior is surprising, however. Pausing from the beginning is probably the right thing to do here, but I'll have to think about how to implement this properly, as it requires the concept of cancelling a previous request.

Also, I'm curious, when you say "which in my opinion is an option that grants a broader set of capabilities", are you referring to "paused from the beginning" or "getting processed all the time"? If the latter, can you explain why you think it's better? It seems to me like processing even all the time would break the back pressure contract of fork.

[pavel]

I referred to "getting processed all the time". My opinion was rushed out and after some thinking I have to say that I don't see any reason for fork function to break back pressure contract.

Though I have to point out that there're cases when consumer-producer relation should work without back pressure, but that can be achieved by just forwarding data from one stream to another with the help of write function.

[vqvu]

Consumer-producer without backpressure is what observe is for. It does exactly what you suggests (forwards data using write).

[pavel]

Forgot about observe. Thanks.