[Feature]: tests take a long time to run locally

[bendichter]

What would you like to see added to nwb-conversion-tools?

Especially the ones in test_move_utils. Can we make these faster?

Is your feature request related to a problem?

No response

What solution would you like?

Especially the ones in test_move_utils. Can we make these faster?

Do you have any interest in helping implement the feature?

No.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this bug was not already reported?

[CodyCBakerPhD]

Looks like test_si.py is the longest set of module tests to run, which we've been aware of for some time and @h-mayorquin was slowly refactoring them in a better fashion.

Also adding here - SpikeInterface recently started using a fancy module-based approach to triggers: https://github.com/SpikeInterface/spikeinterface/blob/master/.github/workflows/full-test.yml#L85-L118

Would be cool if we could do something similar, and changes to things like documentation or changes to differing modalities (an ophys interface vs an ecephys interface) need not trigger tests that only cover things untouched by those changes. Though for safety, should still have daily runs of the full test suite.

[h-mayorquin]

Here is a table of the 100 longest test by duration as they stand now. Organized from the longest to the shortest:

As @CodyCBakerPhD says, some of the biggest culprits are the large (non-unit) tests from the spikeinterface module. There are other large tests like CnmfeSegmentationInterface , CellExplorerSortingInterface and test_pkl_interface.

It seems that making the first 10 tests faster would reduce the total time for local tests greatly as they account for 60 % of the total time.

	time	%total_time	test_name
0	62.15	16.6252	test_write_sorting
1	28	24.1152	test_convert_sorting_extractor_to_nwb_3_CellExplorerSortingInterface_
2	23.28	30.3427	test_write_recording
3	19.22	35.484	test_convert_segmentation_extractor_to_nwb_1_CnmfeSegmentationInterface
4	18.47	40.4248	test_convert_sorting_extractor_to_nwb_4_CellExplorerSortingInterface_
5	15.05	44.4507	test_movie_custom_module
6	13.88	48.1636	test_movie_starting_times
7	12.67	51.5528	test_pkl_interface
8	7.28	53.5003	test_recording_extractor_to_nwb_21_SpikeGLXRecordingInterface_spikeextractors_backend=False
9	7.26	55.4423	test_write_recording_chunking
10	6.31	57.1302	test_convert_sorting_extractor_to_nwb_7_PhySortingInterface_spikeextractors_backend=True
11	5.49	58.5988	test_recording_extractor_to_nwb_03_BlackrockRecordingExtractorInterface_spikeextractors_backend=True
12	5.34	60.0273	test_convert_segmentation_extractor_to_nwb_2_ExtractSegmentationInterface
13	5.04	61.3755	test_recording_extractor_to_nwb_13_SpikeGadgetsRecordingInterface_20210225_em8_minirec2_ac, num_channels=512, gains=0.385spikeextractors_backend=True
14	4.91	62.6889	test_recording_extractor_to_nwb_11_SpikeGadgetsRecordingInterface_20210225_em8_minirec2_ac, num_channels=512, gains=0.195spikeextractors_backend=True
15	4.88	63.9943	test_recording_extractor_to_nwb_09_SpikeGadgetsRecordingInterface_20210225_em8_minirec2_ac, num_channels=512, gains=Nonespikeextractors_backend=True
16	4.6	65.2248	test_run_conversion_from_yaml_default_nwbfile_name
17	4.34	66.3858	test_run_conversion_from_yaml
18	3.88	67.4237	spikeglx.rst
19	3.74	68.4242	test_movie_starting_times
20	3.68	69.4086	test_recording_extractor_to_nwb_04_BlackrockRecordingExtractorInterface_spikeextractors_backend=False
21	3.68	70.393	test_save_movie_to_custom_module
22	3.62	71.3613	test_frame_shape_big
23	2.96	72.1531	test_recording_extractor_to_nwb_14_SpikeGadgetsRecordingInterface_20210225_em8_minirec2_ac, num_channels=512, gains=0.385spikeextractors_backend=False
24	2.96	72.9449	test_convert_sorting_extractor_to_nwb_5_NeuroscopeSortingInterface_
25	2.89	73.718	test_recording_extractor_to_nwb_10_SpikeGadgetsRecordingInterface_20210225_em8_minirec2_ac, num_channels=512, gains=Nonespikeextractors_backend=False
26	2.87	74.4857	test_nwb_metadata
27	2.86	75.2508	test_recording_extractor_to_nwb_12_SpikeGadgetsRecordingInterface_20210225_em8_minirec2_ac, num_channels=512, gains=0.195spikeextractors_backend=False
28	2.8	75.9998	test_recording_extractor_to_nwb_00_NeuralynxRecordingInterface_
29	2.7	76.722	test_custom_chunk_shape
30	2.5	77.3908	test_recording_extractor_to_nwb_05_IntanRecordingInterface_rhd, spikeextractors_backend=True
31	2.49	78.0569	test_recording_extractor_to_nwb_22_SpikeGLXLFPInterface_spikeextractors_backend=True
32	2.48	78.7203	test_recording_extractor_to_nwb_02_AxonaRecordingExtractorInterface_
33	2.4	79.3623	test_recording_extractor_to_nwb_23_SpikeGLXLFPInterface_spikeextractors_backend=False
34	2.38	79.9989	test_recording_extractor_to_nwb_01_OpenEphysRecordingExtractorInterface_
35	2.31	80.6169	test_movie_chunking
36	2.28	81.2268	test_recording_extractor_to_nwb_06_IntanRecordingInterface_rhd, spikeextractors_backend=False
37	2.03	81.7698	test_movie_stub
38	2.02	82.3101	test_convert_sorting_extractor_to_nwb_2_CellExplorerSortingInterface_
39	1.86	82.8077	test_movie_stub
40	1.75	83.2758	test_iterator_stub
41	1.74	83.7413	test_recording_extractor_to_nwb_17_SpikeGadgetsRecordingInterface_W122_06_09_2019_1_fromSD, num_channels=128, gains=0.195spikeextractors_backend=True
42	1.72	84.2014	test_recording_extractor_to_nwb_19_SpikeGadgetsRecordingInterface_W122_06_09_2019_1_fromSD, num_channels=128, gains=0.385spikeextractors_backend=True
43	1.7	84.6561	test_recording_extractor_to_nwb_15_SpikeGadgetsRecordingInterface_W122_06_09_2019_1_fromSD, num_channels=128, gains=Nonespikeextractors_backend=True
44	1.57	85.0761	test_movie_chunking
45	1.52	85.4827	test_movie_no_starting_times_with_exernal_model
46	1.5	85.884	test_convert_imaging_extractor_to_nwb_1_Hdf5ImagingInterface
47	1.48	86.2799	intan.rst
48	1.47	86.6731	test_movie_external_mode
49	1.43	87.0556	test_recording_extractor_to_nwb_07_IntanRecordingInterface_rhs, spikeextractors_backend=True
50	1.42	87.4355	test_movie_irregular_timestamps
51	1.4	87.81	test_make_or_load_nwbfile_overwrite
52	1.4	88.1845	test_external_mode_assertion_with_movie_name_duplication
53	1.39	88.5563	test_make_or_load_nwbfile_append
54	1.37	88.9228	test_movie_duplicate_names_with_external_mode
55	1.33	89.2785	test_tutorials
56	1.31	89.629	test_run_conversion_from_yaml_no_nwbfile_name_or_other_metadata_assertion
57	1.3	89.9767	test_convert_imaging_extractor_to_nwb_0_TiffImagingInterface
58	1.27	90.3165	test_recording_extractor_to_nwb_20_SpikeGadgetsRecordingInterface_W122_06_09_2019_1_fromSD, num_channels=128, gains=0.385spikeextractors_backend=False
59	1.27	90.6562	test_movie_regular_timestamps
60	1.26	90.9932	test_frame_shape_small
61	1.26	91.3303	test_recording_extractor_to_nwb_08_IntanRecordingInterface_rhs, spikeextractors_backend=False
62	1.26	91.6673	test_movie_no_starting_times
63	1.22	91.9937	test_tutorial_interfaces
64	1.19	92.312	blackrock.rst
65	1.17	92.625	test_recording_extractor_to_nwb_16_SpikeGadgetsRecordingInterface_W122_06_09_2019_1_fromSD, num_channels=128, gains=Nonespikeextractors_backend=False
66	1.16	92.9353	test_recording_extractor_to_nwb_18_SpikeGadgetsRecordingInterface_W122_06_09_2019_1_fromSD, num_channels=128, gains=0.195spikeextractors_backend=False
67	1.02	93.2081	test_convert_imaging_extractor_to_nwb_3_SbxImagingInterface
68	0.98	93.4703	test_convert_sorting_extractor_to_nwb_1_BlackrockSortingExtractorInterface_
69	0.95	93.7244	test_convert_segmentation_extractor_to_nwb_0_CaimanSegmentationInterface
70	0.92	93.9705	test_different_channel_properties
71	0.9	94.2113	test_convert_abf_to_nwb_0_AbfInterface
72	0.88	94.4467	test_convert_imaging_extractor_to_nwb_2_SbxImagingInterface
73	0.85	94.674	test_convert_sorting_extractor_to_nwb_6_PhySortingInterface_spikeextractors_backend=False
74	0.84	94.8988	test_convert_segmentation_extractor_to_nwb_3_Suite2pSegmentationInterface
75	0.83	95.1208	test_group_set_custom_description
76	0.83	95.3428	test_convert_sorting_extractor_to_nwb_0_KilosortSortingInterface_
77	0.83	95.5648	openephys.rst
78	0.83	95.7869	test_recording_extractor_to_nwb_25_NeuroscopeRecordingInterface_spikeextractors_backend=False
79	0.82	96.0062	test_make_or_load_nwbfile_write
80	0.82	96.2256	test_neuroscope_gains_0
81	0.8	96.4396	test_append_same_properties
82	0.8	96.6536	test_external_mode
83	0.8	96.8676	kilosort.rst
84	0.79	97.0789	phy.rst
85	0.78	97.2875	test_make_or_load_nwbfile_closure
86	0.78	97.4962	test_convert_lfp_to_nwb_0_AxonaLFPDataInterface_
87	0.78	97.7048	test_movie_starting_times_with_duplicate_names
88	0.77	97.9108	test_sorting_stub
89	0.76	98.1141	test_external_model_with_duplicate_movies
90	0.76	98.3174	test_neuroscope_dtype_0
91	0.76	98.5207	neuroscope.rst
92	0.75	98.7213	axona.rst
93	0.74	98.9193	neuralynx.rst
94	0.7	99.1065	test_recording_extractor_to_nwb_24_NeuroscopeRecordingInterface_spikeextractors_backend=True
95	0.69	99.2911	test_neuroscope_gains_1
96	0.67	99.4703	test_make_or_load_nwbfile_pass_nwbfile
97	0.66	99.6469	test_neuroscope_starting_time
98	0.66	99.8234	cellexplorer.rst
99	0.66	100	test_neuroscope_dtype_1