Delete should prevent orphans in externals and IntervalList

[xlsun79]

After finding errors in my metadata file, I deleted the _.nwb file, raw data .nwb file under stelmo/nwb/raw, as well as the corresponding entry on the Nwbfile() table for the dataset "Lewis20240222". I was able to recreate an nwb raw file with the correct metadata file, but when I was trying to reinsert the raw data into the Nwbfile() table via

import spyglass.data_import as sgi
nwb_file_name =  "Lewis20240222.nwb"
sgi.insert_sessions(nwb_file_name)

It gives the following error:

Error Stack

[12:05:01][INFO] Spyglass: Creating a copy of NWB file Lewis20240222.nwb with link to raw ephys data: Lewis20240222_.nwb
/home/xulu/anaconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-common' version 1.8.0 because version 1.5.1 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/home/xulu/anaconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'core' version 2.6.0-alpha because version 2.5.0 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
/home/xulu/anaconda3/envs/spyglass/lib/python3.9/site-packages/hdmf/spec/namespace.py:531: UserWarning: Ignoring cached namespace 'hdmf-experimental' version 0.5.0 because version 0.2.0 is already loaded.
  warn("Ignoring cached namespace '%s' version %s because version %s is already loaded."
---------------------------------------------------------------------------
DataJointError                            Traceback (most recent call last)
Cell In [6], line 1
----> 1 sgi.insert_sessions(nwb_file_name)

File ~/code/spyglass/src/spyglass/data_import/insert_sessions.py:68, in insert_sessions(nwb_file_names)
     64 # Make a copy of the NWB file that ends with '_'.
     65 # This has everything except the raw data but has a link to
     66 # the raw data in the original file
     67 copy_nwb_link_raw_ephys(nwb_file_name, out_nwb_file_name)
---> 68 Nwbfile().insert_from_relative_file_name(out_nwb_file_name)
     69 populate_all_common(out_nwb_file_name)

File ~/code/spyglass/src/spyglass/common/common_nwbfile.py:75, in Nwbfile.insert_from_relative_file_name(cls, nwb_file_name)
     73 key["nwb_file_name"] = nwb_file_name
     74 key["nwb_file_abs_path"] = nwb_file_abs_path
---> 75 cls.insert1(key, skip_duplicates=True)

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:334, in Table.insert1(self, row, **kwargs)
    327 def insert1(self, row, **kwargs):
    328     """
    329     Insert one data record into the table. For ``kwargs``, see ``insert()``.
    330 
    331     :param row: a numpy record, a dict-like object, or an ordered sequence to be inserted
    332         as one row.
    333     """
--> 334     self.insert((row,), **kwargs)

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:409, in Table.insert(self, rows, replace, skip_duplicates, ignore_extra_fields, allow_direct_insert)
    406     return
    408 field_list = []  # collects the field list from first row (passed by reference)
--> 409 rows = list(
    410     self.__make_row_to_insert(row, field_list, ignore_extra_fields)
    411     for row in rows
    412 )
    413 if rows:
    414     try:

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:410, in <genexpr>(.0)
    406     return
    408 field_list = []  # collects the field list from first row (passed by reference)
    409 rows = list(
--> 410     self.__make_row_to_insert(row, field_list, ignore_extra_fields)
    411     for row in rows
    412 )
    413 if rows:
    414     try:

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:896, in Table.__make_row_to_insert(self, row, field_list, ignore_extra_fields)
    894 elif isinstance(row, collections.abc.Mapping):  # dict-based
    895     check_fields(row)
--> 896     attributes = [
    897         self.__make_placeholder(name, row[name], ignore_extra_fields)
    898         for name in self.heading
    899         if name in row
    900     ]
    901 else:  # positional
    902     try:

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:897, in <listcomp>(.0)
    894 elif isinstance(row, collections.abc.Mapping):  # dict-based
    895     check_fields(row)
    896     attributes = [
--> 897         self.__make_placeholder(name, row[name], ignore_extra_fields)
    898         for name in self.heading
    899         if name in row
    900     ]
    901 else:  # positional
    902     try:

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/table.py:858, in Table.__make_placeholder(self, name, value, ignore_extra_fields)
    852         value = (
    853             str.encode(attachment_path.name)
    854             + b"\0"
    855             + attachment_path.read_bytes()
    856         )
    857 elif attr.is_filepath:
--> 858     value = self.external[attr.store].upload_filepath(value).bytes
    859 elif attr.numeric:
    860     value = str(int(value) if isinstance(value, bool) else value)

File ~/anaconda3/envs/spyglass/lib/python3.9/site-packages/datajoint/external.py:279, in ExternalTable.upload_filepath(self, local_filepath)
    276 if check_hash:
    277     # the tracking entry exists, check that it's the same file as before
    278     if contents_hash != check_hash[0]:
--> 279         raise DataJointError(
    280             f"A different version of '{relative_filepath}' has already been placed."
    281         )
    282 else:
    283     # upload the file and create its tracking entry
    284     self._upload_file(
    285         local_filepath,
    286         self._make_external_filepath(relative_filepath),
    287         metadata={"contents_hash": str(contents_hash)},
    288     )

DataJointError: A different version of 'Lewis20240222_.nwb' has already been placed.

How could I clean up all traces of creating the older, incorrect nwb file besides deleting the corresponding raw data & table? Thanks!

[CBroz1]

This looks to me like the externals table still has a record of this file. You can access it with the following, and then delete the entry. This delete is currently run during the nightly cleanup

from {module} fetch schema
(schema.externals['analysis'] & your_restriction)

A future version of Spyglass could use the delete method to anticipate these deletes and include them whenever a delete is performed on a table with a fk ref to AnalysisNwbfile. Similarly with IntervalList orphans.

[xlsun79]

I think I finally figured what may be going on -- after deleting the entry from the Nwbfile table, I should also do a cleanup of the file paths. I was able to reinsert the data after that =)