Failure runs: <a href="https://github.com/zhuyifei1999/guppy3/

Seems non-deterministic for some reason <a href="https://github.com/zhuyifei1999/guppy

Test passing again after <a class="commit-link" data-hovercard-type="commit" data-hove

Heisenbug: test_RefPat.RefPatCase.test_presentation fails sometimes on Python 3.9 on Windows about guppy3 HOT 4 CLOSED

zhuyifei1999 commented on July 18, 2024

Heisenbug: test_RefPat.RefPatCase.test_presentation fails sometimes on Python 3.9 on Windows

from guppy3.

Comments (4)

zhuyifei1999 commented on July 18, 2024

Seems non-deterministic for some reason https://github.com/zhuyifei1999/guppy3/runs/4731656956?check_suite_focus=true observes failure only when both coverage and sdist are disabled, whereas most of my attempts show the exact reverse.

Targeting the same code to all targets got a new reproduction on ubuntu Python 3.9, with both coverage and sdist disabled: https://github.com/zhuyifei1999/guppy3/runs/4731739323?check_suite_focus=true

from guppy3.

zhuyifei1999 commented on July 18, 2024

In this recent PR the target that is failing has changed to Python 3.10 for some wild reason. #43 Also seems to happen when codecov = true

from guppy3.

zhuyifei1999 commented on July 18, 2024

Test passing again after 6e1d780 & 9c9dccc

I'm honestly wondering if it has something to do with being an ediable install and the big warning banner:

          ********************************************************************************
          Please be careful with folders in your working directory with the same
Successfully built guppy3
          name as your package as they may take precedence during imports.
          ********************************************************************************

  !!

Even though the test script is in /tmp, and /tmp/test.py seems unable to import CWD without installing: https://github.com/zhuyifei1999/guppy3/actions/runs/4970663631/jobs/8894682618, there may still be some sort of shenanigans with import path going on.

from guppy3.

zhuyifei1999 commented on July 18, 2024

Figured out the root cause. RefPat calls a set's .referrers, and the way it works is subject to a race with the GC, which can cause it to unintentionally return an empty set.

The logic of referrers performs operation on the reference graph (rg):

guppy3/guppy/heapy/View.py

Lines 456 to 496 in 5a35ccb

 def referrers(self, X): 

 """V.referrers(X) -> idset 

 Return the set of objects that directly refer to 

 any of the objects in the set X.""" 

 X = self.nodeset_adapt(X) 

 if self.is_rg_update_all and self.root is self.heapyc.RootState: 

 if not (self.rg.domain_covers(X) or 

 self.rg.domain_covers(X - self.norefer)): 

 self.rg.clear() 

 import gc 

 gc.collect() 

 self.hv.update_referrers_completely(self.rg) 

 addnoref = X - self.rg.get_domain() 

 self.norefer |= addnoref 

 else: 

 Y = self.mutnodeset(X) 

 Y -= self.norefer 

 if not self.rg.domain_covers(Y): 

 for wt in self.referrers_targets: 

 t = wt() 

 if t is not None: 

 Y |= t.set.nodes 

 Y |= self.rg.get_domain() 

 self.rg.clear() 

 self.hv.update_referrers(self.rg, Y) 

 self.norefer.clear() 

 self.norefer |= (X | Y | self.rg.get_range()) 

 self.norefer -= self.rg.get_domain() 

 Y = self.mutnodeset(X) - self.norefer 

 if not self.rg.domain_covers(Y): 

 print('update_referrers failed') 

 print('Y - domain of rg:') 

 print(self.idset(Y - self.rg.get_domain())) 

 Y = None 

 X = self.rg.relimg(X) 

 X = self.immnodeset(X) - [None] 

 X = self.retset(X) 

 return X

The reference graph is populated at line 481, but is used at 493. If between this time, the cyclic GC runs, we have:

guppy3/guppy/heapy/View.py

Lines 107 to 119 in 5a35ccb

 def clear_callback(self, wr): 

 self._clear_hook.clear() 

 for m in self.clear_methods: 

 m() 

 self.clear_setup() 

 def clear_setup(self): 

 ch = self._clear_hook 

 ch.clear() 

 c = self.gchook_type() 

 cb = self.ClearCallback(self.clear_callback) 

 c.cb = cb 

 ch.add(self._root.weakref.ref(c, cb))

Here c is a cyclic object, so clear_callback gets called whenever GC runs, and what's inside clear_methods?

guppy3/guppy/heapy/View.py

Lines 124 to 126 in 5a35ccb

 def clear_register_method(self, m): 

 self.clear_methods.append(m) 

 self.clear_check()

guppy3/guppy/heapy/View.py

Lines 177 to 190 in 5a35ccb

 def _get_rg(self): 

 rg = self.nodegraph() 

 self.clear_register_method(self._clear_rg) 

 return rg 

 def _clear_rg(self): 

 if self.referrers_lock: 

 return 

 rg = self.rg 

 if rg.is_sorted: 

 rg.clear() 

 self.norefer.clear() 

 else: 

 pass

The reference graph is cleared upon GC. This makes sense, because reference graph holds strong references to everything it contains, and we don't want objects staying alive forever just because guppy has seen them.

However this is a problem, because if GC runs between the populating of the reference graph and the usage of the reference graph, reference graph would now be empty when it's used, and ... we return bogus results.

This theory is tested in commit 6c7c84d. I'll now port it to master branch.

from guppy3.

Heisenbug: test_RefPat.RefPatCase.test_presentation fails sometimes on Python 3.9 on Windows about guppy3 HOT 4 CLOSED

Comments (4)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	def referrers(self, X):
	"""V.referrers(X) -> idset
	Return the set of objects that directly refer to
	any of the objects in the set X."""

	X = self.nodeset_adapt(X)
	if self.is_rg_update_all and self.root is self.heapyc.RootState:
	if not (self.rg.domain_covers(X) or
	self.rg.domain_covers(X - self.norefer)):
	self.rg.clear()
	import gc
	gc.collect()
	self.hv.update_referrers_completely(self.rg)
	addnoref = X - self.rg.get_domain()
	self.norefer \|= addnoref
	else:
	Y = self.mutnodeset(X)
	Y -= self.norefer
	if not self.rg.domain_covers(Y):
	for wt in self.referrers_targets:
	t = wt()
	if t is not None:
	Y \|= t.set.nodes
	Y \|= self.rg.get_domain()
	self.rg.clear()
	self.hv.update_referrers(self.rg, Y)
	self.norefer.clear()
	self.norefer \|= (X \| Y \| self.rg.get_range())
	self.norefer -= self.rg.get_domain()
	Y = self.mutnodeset(X) - self.norefer
	if not self.rg.domain_covers(Y):
	print('update_referrers failed')
	print('Y - domain of rg:')
	print(self.idset(Y - self.rg.get_domain()))

	Y = None

	X = self.rg.relimg(X)
	X = self.immnodeset(X) - [None]
	X = self.retset(X)
	return X

	def clear_callback(self, wr):
	self._clear_hook.clear()
	for m in self.clear_methods:
	m()
	self.clear_setup()

	def clear_setup(self):
	ch = self._clear_hook
	ch.clear()
	c = self.gchook_type()
	cb = self.ClearCallback(self.clear_callback)
	c.cb = cb
	ch.add(self._root.weakref.ref(c, cb))

	def clear_register_method(self, m):
	self.clear_methods.append(m)
	self.clear_check()

	def _get_rg(self):
	rg = self.nodegraph()
	self.clear_register_method(self._clear_rg)
	return rg

	def _clear_rg(self):
	if self.referrers_lock:
	return
	rg = self.rg
	if rg.is_sorted:
	rg.clear()
	self.norefer.clear()
	else:
	pass