Improvement to the `test_spot_target_excludes` test. #6652

drivanov · 2023-12-01T00:38:56Z

Description

The main goal of these changes is the elimination of the following 15 warnings:

The main purpose of these changes is to address the following 15 warnings:
  /usr/local/lib/python3.10/dist-packages/dgl/dataloading/dataloader.py:1149: DGLWarning: Dataloader CPU affinity opt is not enabled, consider switching it on (see enable_cpu_affinity() or CPU best practices for DGL [https://docs.dgl.ai/tutorials/cpu/cpu_best_practises.html])
    dgl_warning(

generated by the test_spot_target_excludes test. In addition to this, we have extended the list of variable input parameters with num_workers with values of 1 and 4.

Checklist

Please feel free to remove inapplicable items for your PR.

I've leverage the tools to beautify the python and c++ code.
The PR is complete and small, read the Google eng practice (CL equals to PR) to understand more about small PR. In DGL, we consider PRs with less than 200 lines of core code change are small (example, test and documentation could be exempted).
To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

dgl-bot · 2023-12-01T00:39:22Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-01T00:39:34Z

Commit ID: 799256cf9001183ac607dd2215a0fc2cf157321d

Build ID: 1

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-01T01:44:48Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-01T01:45:00Z

Commit ID: a279d3315cde882a7c0ea0a5d98f3fc4276435fb

Build ID: 2

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-01T17:31:05Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-01T17:31:17Z

Commit ID: 8a236bc

Build ID: 3

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-09T20:00:46Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-09T20:00:57Z

Commit ID: 6894c64

Build ID: 4

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-11T16:25:49Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-11T16:26:01Z

Commit ID: 363fdf4

Build ID: 5

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-11T19:34:31Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-11T19:34:43Z

Commit ID: d340cc1

Build ID: 6

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-13T00:41:15Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-13T00:41:27Z

Commit ID: ce67fe9

Build ID: 7

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-13T17:01:00Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-13T17:01:12Z

Commit ID: faae457

Build ID: 8

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-20T22:40:36Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-20T22:40:49Z

Commit ID: f0dc7a6

Build ID: 9

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2023-12-30T20:55:05Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2023-12-30T20:55:17Z

Commit ID: c874e95

Build ID: 10

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2024-01-10T19:21:40Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-01-10T19:21:52Z

Commit ID: 6612adf

Build ID: 16

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2024-01-17T01:23:58Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-01-17T01:24:10Z

Commit ID: 49ee714

Build ID: 17

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2024-01-19T19:53:32Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-01-19T19:53:45Z

Commit ID: 0f053e6

Build ID: 18

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2024-01-22T20:12:04Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-01-22T20:12:16Z

Commit ID: fb46bb4

Build ID: 19

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2024-01-28T02:27:53Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-01-28T02:28:05Z

Commit ID: 99e10df

Build ID: 20

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

dgl-bot · 2024-02-01T23:13:01Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-02-01T23:13:13Z

Commit ID: 83fa16a

Build ID: 21

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

frozenbugs · 2024-02-09T02:43:50Z

@dgl-bot

dgl-bot · 2024-02-09T03:30:42Z

Commit ID: 1cae14f54e5253d1d1321c814b430e38036b1fcc

Build ID: 22

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

dgl-bot · 2024-02-09T17:16:30Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-02-09T17:16:42Z

Commit ID: b3a66eb

Build ID: 23

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

frozenbugs · 2024-02-19T02:10:15Z

@dgl-bot

frozenbugs · 2024-02-19T02:12:23Z

@drivanov

================================== FAILURES ===================================
______________________ test_spot_target_excludes[1-1-1] _______________________

degree_threshold = 1, batch_size = 1, num_workers = 1

    @pytest.mark.parametrize("degree_threshold", [1, 2, 3, 4, 5])
    @pytest.mark.parametrize("batch_size", [1, 10, 50])
    @pytest.mark.parametrize("num_workers", [1, 4])
    def test_spot_target_excludes(degree_threshold, batch_size, num_workers):
        g, reverse_eids, seed_edges = _create_homogeneous()
        sampler = dgl.dataloading.MultiLayerFullNeighborSampler(1)
        low_degree_excluder = dgl.dataloading.SpotTarget(
            g,
            exclude="reverse_id",
            degree_threshold=degree_threshold,
            reverse_eids=reverse_eids,
        )
        sampler = dgl.dataloading.as_edge_prediction_sampler(
            sampler,
            exclude=low_degree_excluder,
            negative_sampler=dgl.dataloading.negative_sampler.Uniform(1),
        )
        dataloader = dgl.dataloading.DataLoader(
            g,
            seed_edges,
            sampler,
            batch_size=batch_size,
            num_workers=num_workers,
        )
    
        with dataloader.enable_cpu_affinity():
            for i, (input_nodes, pair_graph, neg_pair_graph, blocks) in enumerate(
&gt;               dataloader
            ):

tests\python\pytorch\dataloading\test_spot_target.py:58: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
python\dgl\dataloading\dataloader.py:1160: in __iter__
    self, super().__iter__(), num_threads=num_threads
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py:435: in __iter__
    return self._get_iterator()
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py:381: in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py:1034: in __init__
    w.start()
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py:112: in start
    self._popen = self._Popen(self)
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py:223: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py:322: in _Popen
    return Popen(process_obj)
C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py:89: in __init__
    reduction.dump(process_obj, to_child)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = &lt;Process(Process-117, initial daemon)&gt;
file = &lt;_io.BufferedWriter name=11&gt;, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
&gt;       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'DataLoader.enable_cpu_affinity.&lt;locals&gt;.init_fn'

C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py:60: AttributeError
---------------------------- Captured stdout call -----------------------------
1 DL workers are assigned to cpus [0], main process will use cpus [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
-------------------------- Captured stderr teardown ---------------------------
Traceback (most recent call last):

  File "&lt;string&gt;", line 1, in &lt;module&gt;

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main

    exitcode = _main(fd)

  File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 115, in _main

    self = reduction.pickle.load(from_parent)

EOFError: Ran out of input

dgl-bot · 2024-02-19T02:57:00Z

Commit ID: 345d45e1573b024d1d6d027dd751e58508c2e0cf

Build ID: 24

Status: ❌ CI test failed in Stage [Torch CPU (Win64) Unit test].

Report path: link

Full logs path: link

drivanov · 2024-02-20T19:45:44Z

@frozenbugs : Unfortunately, I am unable to reproduce the problem you mentioned in my Linux environment. This appears to be some kind of multi-threading bug that only appears on Windows. Similar problems were reported for PR#6187 and PR#6194 where I suggested similar fixes. I'll convert this to "Draft".

dgl-bot · 2024-02-20T19:47:36Z

Not authorized to trigger CI. Please ask core developer to help trigger via issuing comment:

@dgl-bot

dgl-bot · 2024-02-20T19:47:48Z

Commit ID: 2767e32460e234000332f427cc3ae6f3553eed9c

Build ID: 25

Status: ❌ CI test failed in Stage [Authentication].

Report path: link

Full logs path: link

Improvement to the test_spot_target_excludes test.

d5c3d96

Fixing lint problems.

b5a9d92

Merge branch 'master' into spot_target

8a236bc

Merge branch 'master' into spot_target

6894c64

Merge branch 'master' into spot_target

363fdf4

Merge branch 'master' into spot_target

d340cc1

Merge branch 'master' into spot_target

ce67fe9

Merge branch 'master' into spot_target

faae457

Merge branch 'master' into spot_target

f0dc7a6

Merge branch 'master' into spot_target

c874e95

Merge branch 'master' into spot_target

49ee714

Merge branch 'master' into spot_target

0f053e6

Merge branch 'master' into spot_target

fb46bb4

Merge branch 'master' into spot_target

99e10df

Merge branch 'master' into spot_target

83fa16a

frozenbugs self-requested a review February 9, 2024 02:43

Merge branch 'master' into spot_target

b3a66eb

drivanov marked this pull request as draft February 20, 2024 19:47

Improvement to the test_spot_target_excludes test. #6652

Are you sure you want to change the base?

Improvement to the test_spot_target_excludes test. #6652

Conversation

drivanov commented Dec 1, 2023

Description

Checklist

dgl-bot commented Dec 1, 2023

dgl-bot commented Dec 1, 2023

dgl-bot commented Dec 1, 2023

dgl-bot commented Dec 1, 2023

dgl-bot commented Dec 1, 2023

dgl-bot commented Dec 1, 2023

dgl-bot commented Dec 9, 2023

dgl-bot commented Dec 9, 2023

dgl-bot commented Dec 11, 2023

dgl-bot commented Dec 11, 2023

dgl-bot commented Dec 11, 2023

dgl-bot commented Dec 11, 2023

dgl-bot commented Dec 13, 2023

dgl-bot commented Dec 13, 2023

dgl-bot commented Dec 13, 2023

dgl-bot commented Dec 13, 2023

dgl-bot commented Dec 20, 2023

dgl-bot commented Dec 20, 2023

dgl-bot commented Dec 30, 2023

dgl-bot commented Dec 30, 2023

dgl-bot commented Jan 10, 2024

dgl-bot commented Jan 10, 2024

dgl-bot commented Jan 17, 2024

dgl-bot commented Jan 17, 2024

dgl-bot commented Jan 19, 2024

dgl-bot commented Jan 19, 2024

dgl-bot commented Jan 22, 2024

dgl-bot commented Jan 22, 2024

dgl-bot commented Jan 28, 2024

dgl-bot commented Jan 28, 2024

dgl-bot commented Feb 1, 2024

dgl-bot commented Feb 1, 2024

frozenbugs commented Feb 9, 2024

dgl-bot commented Feb 9, 2024

dgl-bot commented Feb 9, 2024

dgl-bot commented Feb 9, 2024

frozenbugs commented Feb 19, 2024

frozenbugs commented Feb 19, 2024 • edited

dgl-bot commented Feb 19, 2024

drivanov commented Feb 20, 2024

dgl-bot commented Feb 20, 2024

dgl-bot commented Feb 20, 2024

Improvement to the `test_spot_target_excludes` test. #6652

Improvement to the `test_spot_target_excludes` test. #6652

frozenbugs commented Feb 19, 2024 •

edited