key error - on healthy cluster

[dthpulse]

Hi,
trying to run with options

 -v balance --osdsize device --osdused delta --max-pg-moves 2 --osdfrom fullest

getting error:

[2024-04-15 12:31:58,685] gathering cluster state via ceph api...
[2024-04-15 12:32:33,852] running pg balancer
Traceback (most recent call last):
  File "jj.py", line 5496, in <module>
    exit(main())
  File "jj.py", line 5490, in main
    run()
  File "jj.py", line 5454, in <lambda>
    run = lambda: balance(args, state)
  File "jj.py", line 4607, in balance
    pg_mappings = PGMappings(cluster,
  File "jj.py", line 3265, in __init__
    self.init_analyzer.analyze(self)
  File "jj.py", line 4288, in analyze
    self._update_stats()
  File "jj.py", line 4374, in _update_stats
    self.cluster_variance = self.pg_mappings.get_cluster_variance()
  File "jj.py", line 3788, in get_cluster_variance
    for crushclass, usages in self.get_class_osd_usages().items():
  File "jj.py", line 3526, in get_class_osd_usages
    ret[crushclass] = {osdid: self.get_osd_usage(osdid) for osdid in self.osd_candidates_class[crushclass]}
  File "jj.py", line 3526, in <dictcomp>
    ret[crushclass] = {osdid: self.get_osd_usage(osdid) for osdid in self.osd_candidates_class[crushclass]}
  File "jj.py", line 3774, in get_osd_usage
    used = self.get_osd_usage_size(osdid, add_size)
  File "jj.py", line 3731, in get_osd_usage_size
    used += self.cluster.osd_transfer_remainings[osdid]
KeyError: 3

Cluster is Ceph Quincy, with 1500 OSDs, SSDs, HDDs, with more pools, but main and most utilized is EC.
Any other details you need?

Thank you