runaway memory growth during GC with reference cycle about ponyc HOT 12 CLOSED

@mfelsche can you highlight the "you can detect the repetition in pointer values" to make sure what you mean is the same as what others looking at the stack would also mean?

from ponyc.

Can you clarify "anymore" and when specifically in terms of pony version this arose?

from ponyc.

Do you have a minimal example that is runnable by others? The smaller example does not compile so its impossible to debug.

from ponyc.

The ponyc version is the current tip of the main branch: commit 4ec27c827ab32d73937db198bfcae4be4733dcef. Didnt try on older versions yet.

Sorry for the bad reproducer, i fixed it to compile. Typo on my end.

After the create_reference_cycle behavior is done executing, the runtime seems to be stuck in doing gc.

from ponyc.

This issue exists in pony 0.50.0 so it is at least several months old. I took "anymore" to be "recently worked".

I'm going to edit the original comment.

from ponyc.

I don't think this is a bug, but rather expected behavior!

The code creates an endless loop inside a behavior. However, the runtime attempts GC cycles only at the end of behaviors. The code prevents this from ever happening since the loop variable j is never changed from its initial value 10. Therefore, the “run-away” memory allocation is bound to happen.

EDIT: This is also discussed here. What is done in the code is practically the same as the giant for loop in the example there...
https://tutorial.ponylang.io/gotchas/garbage-collection.html#long-running-behaviors-and-memory

from ponyc.

OMG, I can't believe I missed that. Thanks @stefandd. I'll do a test without the endless loop to verify all works as expected.

from ponyc.

@SeanTAllen I just tried it with if j > 0 then instead of while j > 0 do in the playground, and it finishes fine with Done

from ponyc.

Yeah I tried a variation on this as well with a timer and I don't see an indication of a leak. I can see that nodes are getting collected with variations on this program that have a _final on Node.

use "collections"
use "time"

class Node

  var parent: (Node | None) = None
  var child: (Node | None) = None

  new ref create() => None

actor Main
  let _timers: Timers
  var ast: (Node | None) = None

  be create_reference_cycle() =>
    let root = Node
    let child = Node
    child.parent = root

    // add children pointing back at parent
    var j: USize = 10
    var up = child
    while j > 0 do
      let down = Node
      down.parent = up
      up = down
      j = j - 1
    end
    ast = root

  new create(env: Env) =>
    _timers = Timers
    let timer = Timer(Notify(this), 5_000, 2_000)
    _timers(consume timer)

  class Notify is TimerNotify
    let _main: Main

    new iso create(main: Main) =>
      _main = main

    fun ref apply(timer: Timer, count: U64): Bool =>
      _main.create_reference_cycle()
      true

from ponyc.

Sorry for the bad reproducer.

But I have definitely seen this behaviour in the wild with ponyc 0.53. I came across it when modelling the ponyc ast_t type as a struct AST in pony with the parent and child pointers being NullablePointer[AST]. The ast_t/AST was created by libponyc and then returned as a plain AST struct. It was creating trouble when the gc kicked in.

I worked around it by not modelling the ast_t as a pony struct at all. I will try to find a simple reproducer for this with structs. And check again if it reproduces in the first place. Will reopen if so.

from ponyc.

@mfelsche Just curious - does this mean that are in essence two versions of the Pony GC running concurrently in your use case? (the one in your Pony program and the one in the wrapped libponyc) I wonder if GC troubles stem from such linked objects being traced by two GC instances?

from ponyc.

No. There wouldn't be two versions of the garbage collector.

from ponyc.