Stack overflow with recursive job

[Dolfik1]

I got stack overflow exception with this code:

  let tcs = TaskCompletionSource()
  let rec loop i =
    job {
      if i < 1_000_000 then
        return! loop (i + 1)
      else
        tcs.SetResult()
    }
    
  loop 0 |> start
  tcs.Task.Wait()

This code works fine with 100k iterations but fails on 1kk iterations.

I tried replacing recursion with a loop:

    let tcs = TaskCompletionSource()
    let loop i =
      job {
        let mutable i = 0
        while true do
          if i >= 100_000 then
            tcs.SetResult()
          i <- i + 1
      }
    
    loop 0 |> start
    tcs.Task.Wait()

But got exception on 100k iterations. Same code works fine wtih 10k iterations.

[Dolfik1]

I also got stack overflow on this code:

let task =
  async {
    
    let countAgents = 1
    let countMessages = 1_000_000
    
    let loop (mailbox: Hopac.Mailbox<_>) =
      job {
        match! mailbox with
        | Message.Done tcs ->
          tcs.SetResult(())
        | _ -> ()
      }
      |> Job.foreverServer
      |> start

    let agents =
      [|
        for _ in 1 .. countAgents do
          let processor = Hopac.Mailbox()
          loop processor
          processor
      |]
    
    let random = Random countMessages
    
    for _ = 1 to countMessages do
      let t = random.Next(0, countAgents - 1)
      let mb = agents.[t]
      Mailbox.send mb Message.Unit |> start
      
    let tcs =
      [|
        for i in 0 .. countAgents - 1 do
          let t = TaskCompletionSource()
          Mailbox.send agents.[i] (Message.Done t) |> start
          
          t.Task
      |]
    
    do! Task.WhenAll(tcs) |> Async.AwaitTask |> Async.Ignore
  } |> Async.StartAsTask
task.Wait()

Stack overflow goes away when countAgents > 3

[haf]

Hi, good reporting. I think you should try this both in Debug and Release builds and with and without tail optimisation enabled for the F# compiler, and report back. It seems it has more to do with the optimisations applied to your program than how Hopac is built, however. If you can create a repro, with all optimisations applied, and we can reproduce it, it's a useful debugging target.

[Dolfik1]

I successfully reproduced bug with all examples above in Debug and Release configuration with and without tailcall optimization. You can try to paste any code block below into main function and get same result.

[haf]

Ok, great! Currently I'm not working much with F#, but I'll review and discuss PR:s if someone brings them forward.

[potrykus]

FWIW I also got a stack overflow on a distinct type of recursive server Job.

No stack overflow Alt.choosy, Yes stack overflow Alt.chooser. Too many random numbers on the stack?

`

let altconsumeserver (consumer: 'a option [] -> unit) (sss: Stream<'a option> []) = 
    // stream array's length is fixed
    let memory = Array.init sss.Length (fun _ -> None)

    let rec server (sss: Promise<Stream.Cons<int * Option<'a>>> []): Job<unit> = //Hopac.Void
        Alt.choosy sss // OK. Stack overflow BUG: Alt.chooser. Length ~ a dozen or two
    ^-> (function
          | Stream.Cons((i, v), rst) ->
                memory.[i] <- v
                consumer memory
                sss.[i] <- rst
                sss
          //| TODO logic to shrink the array if an i is done
        ) 
    >>= server //:> Job<Hopac.Void>
    
    server (Array.mapi (fun i -> Stream.mapFun (fun oa -> i, oa)) sss)

`

Please ignore the non-recursion (array re-use) - that would just be bugs in downstream reads. Not library-ready code, obviously.