The problem with refcount of python object

[woodenwatcher]

1. from functools import lru_cache
2. import fluent.handler
3. import logging
4. import logging.config
5. import sys
6. from concurrent.futures import ThreadPoolExecutor
7. 
8. logging.basicConfig(level=logging.INFO)
9. 
10. fluentd_format = {
11.     'hostname': '%(hostname)s',
12.     'module': '%(module)s.%(funcName)s',
13.     'where': '%(filename)s:%(lineno)d',
14.     'level': '%(levelname)s',
15.     'exception': '%(exc_text)s',
16.     'process': '%(processName)s',
17. }
18. 
19. 
20. def fluentd_handler(pendings):
21.     if fluentd_handler.first_time:
22.         fluentd_handler.first_time = False
23.         print("failed to send log to fluentd")
24. 
25. 
26. fluentd_handler.first_time = True
27. 
28. def setup_fluent(tag, host, port):
29.     root = logging.getLogger()
30.     fluent_handler = fluent.handler.FluentHandler(tag, host=host, port=port,
31.                                                   buffer_overflow_handler=fluentd_handler,
32.                                                   nanosecond_precision=True)
33.     fluent_handler.setLevel(logging.INFO)
34.     formatter = fluent.handler.FluentRecordFormatter(fluentd_format)
35.     fluent_handler.setFormatter(formatter)
36.     root.addHandler(fluent_handler)
37. 
38. class TestObj(object):
39.     def __init__(self, name, idx):
40.         self.name = name
41.         self.idx = idx
42. 
43.     def __del__(self):
44.         print("test obj destroyed!")
45. 
46.     def doing(self):
47.         print('name:%s,idx:%s' % (self.name, self.idx))
48. 
49. 
50. @lru_cache(maxsize=64)
51. def get_object(name , id):
52.     return TestObj(name, id);
53. 
54. 
55. def cache_test():
56.     obj1 = get_object('segment000', 0)
57.     logging.info("TestObj's refcount is %s", sys.getrefcount(obj1))
58. 
59. if __name__ == '__main__':
60.     setup_fluent('cachetest', 'localhost', 24224)
61.     pool = ThreadPoolExecutor(max_workers=2)
62.     for i in range(20):
63.         f = pool.submit(cache_test)
64.         f.result()
65.     logging.info('before clearing,cache info:%s', get_object.cache_info())
66.     get_object.cache_clear()
67.     logging.info('after clearing,cache info:%s', get_object.cache_info())

[woodenwatcher]

The refcount of "obj1" (line 57) always increase never decrease but when I delete the code of line 60, the refcount is correct and the destructor of "obj1" was called when clearing cache (line 66). could anyone tell me why?

[woodenwatcher]

@arcivanov

[arcivanov]

Hi @woodenwatcher, thanks for your report.

I'm not part of the Fluent staff and my participation in the project has always been on a purely volunteer basis, i.e. subject to time availability.

Currently I have no time availability.

If you identify the issue and create a pull request I'll be happy to review and merge.
Otherwise you'll need to wait for if and when I have time 😄