Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows 11 or >=R2021b file reading errors #2082

Open
mjcourte opened this issue Jul 28, 2022 · 12 comments · May be fixed by #2383
Open

Windows 11 or >=R2021b file reading errors #2082

mjcourte opened this issue Jul 28, 2022 · 12 comments · May be fixed by #2383

Comments

@mjcourte
Copy link

mjcourte commented Jul 28, 2022

Two errors I have encountered in my Win11 environment. Originally with R2021b, then with R2022a. Fieldtrips 20191213, 20220104, 20220727.

In my Ubuntu 20.04 LTS environment with R2021a, Fieldtrip 20191213, these files all load fine. Thats my old laptop with not enough RAM for the job.

  1. Reading CTF MEG .ds files:
cfg = [];
cfg.continuous = 'yes';
cfg.dataset = ds_C_path;
data_C = ft_preprocessing(cfg);

Returns:

Operands to the logical AND (&&) and OR (||) operators must be convertible to logical scalar values. Use the ANY or
ALL functions to reduce operands to logical scalar values.

Error in ft_filetype (line 176)
if ~contains(filename , '://') && isfolder(filename)

Error in ft_checkconfig (line 720)
    cfg.headerformat = ft_filetype(cfg.headerfile);

Error in ft_preprocessing (line 384)
  cfg = ft_checkconfig(cfg, 'dataset2files', 'yes');
  1. Reading headshape:
headshape = ft_read_headshape(pos_path);

Returns:

Error using exist
The first input to exist must be a string scalar or character vector.

Error in ft_read_headshape (line 207)
  if exist(fullfile(pathstr,[name,'.jpg']), 'file')
  • Data paths in question return true when checking isfile() or isfolder() as approriate.
  • If I run ft_filetype(ds_C_path), it returns 'ctf_ds' as expected.
  • If I run the line in the first error ~contains ... && isfolder ..., on the ds_C_path it returns true.

I'm installing R2021a on my Win11 machine now to see if this reproduces.

@mjcourte
Copy link
Author

Edit to add Win11/R2021a shows the same errors.

@robertoostenveld
Copy link
Contributor

I don't have a Windows 11 machine to test on. Can you try dbstop if error on the command line and run it again? That allows you to look in more detail what the error is.

@schoffelen
Copy link
Contributor

Hi Matt, did you already find time to further look into this?

@mjcourte
Copy link
Author

mjcourte commented Aug 12, 2022

I think I have found the issue. Somewhere along the line, Matlab or Windows, has changed how string concatenation works in Matlab.

In ft_read_header, there is a call
[filename, headerfile, datafile] = dataset2files(filename, headerformat)

Outputs headerfile and datafile are erroneously 1x2 string arrays instead of strings.
They should be path/to/dsname.ds/dsname.res4, and path/to/dsname.ds/dsname.meg4, respectively, but instead they return as

headerfile = [ ...
path/to/dsname.ds/dsname,
path/to/dsname.ds/.res4
]
datafile = [ ...
path/to/dsname.ds/dsname,
path/to/dsname.ds/.meg4
]

The string splits at the ., resulting in an 1x2 array. This throws the next if-statement out because the logic is not expecting arrays as inputs.

@mjcourte
Copy link
Author

mjcourte commented Aug 12, 2022

Specifically, in dataset2files()

...
  case {'ctf_ds', 'ctf_old'}
    % convert CTF filename into filenames
    [path, file, ext] = fileparts(filename);
    if any(strcmp(ext, {'.res4' '.meg4', '.1_meg4' '.2_meg4' '.3_meg4' '.4_meg4' '.5_meg4' '.6_meg4' '.7_meg4' '.8_meg4' '.9_meg4'}))
      filename = path;
      [path, file, ext] = fileparts(filename);
    end
    if isempty(path) && isempty(file)
      % this means that the dataset was specified as the present working directory, i.e. only with '.'
      filename = pwd;
      [path, file, ext] = fileparts(filename);
    end
    headerfile = fullfile(filename, [file '.res4']); % <---- These lines are treating [file '.ext'] as string arrays, not concatting to string.
    datafile   = fullfile(filename, [file '.meg4']); % <----
    if length(path)>3 && strcmp(path(end-2:end), '.ds')
      filename = path; % this is the *.ds directory
    end
...

This change in shorthand string concatenation with string variables has also cropped up elsewhere on my new system in other Matlab work. I'll note that the behaviour has been irritating ,eg.

 msg = ['\tSession no. ' num2str(s) '\n'];
 fprintf(msg)

returns

    Session no. 3

but

msg = ['Tag no. ' tagID];
fprintf(msg)

where tagID var contains a string
returns

Error using fprintf
Invalid file identifier.  Use fopen to generate a valid file identifier.

What happened here is msg was a 1x2 string array, not a concatenated string.

Instead, I had to use +

msg = '\tTag no. ' + tagID + '\n';
fprintf(msg)

returns

	Tag no. 50211

Some quick testing, this seems to not be consistent behaviour. I couldn't say why...

The only helpful suggestion/workaround I have at the moment is to recommend strcat() or append() instead of shorthand [str1 str2].

@schoffelen
Copy link
Contributor

OK, thanks for looking into it. I think that the solution with the '+' is not robust for older matlab versions, and I don't know about append(). We may need to consider to change the square bracket syntax into strcat() operations to stay future proof (and maintain backward compatibility). Oh my...

BTW, your pasted solution with the fullfile() in dataset2files was your doing, right? Would it be possible to file a PR for that?

@schoffelen
Copy link
Contributor

BTW >=R2021b on Linux seems to run fine in my experience.

@schoffelen
Copy link
Contributor

Hi @mjcourte I think the below behavior reproduces the basis of this issue.

If I do (on matlab2021a):

s = ['a' 'b'], I get 'ab'

if I do:

s = ["a" "b"], I get a 1x2 string array.

Long story short, the FT syntax has not been updated to be robust against a mixed usage of chars and the newer 'string' class in matlab

@mjcourte
Copy link
Author

mjcourte commented Aug 16, 2022

@schoffelen

BTW, your pasted solution with the fullfile() in dataset2files was your doing, right? Would it be possible to file a PR for that?

This is from one of the old version of FT that I had lying around, 20190709. I tested this with 20220104, 20220727 as well and they had the same issues, however I did not debug or see the diff in dataset2files.

s = ['a' 'b'], I get 'ab'

if I do:

s = ["a" "b"], I get a 1x2 string array.

To complicate this, the version of dataset2files that has [path 'ext'] is returning the string arrays on my Windows 11 machine, even with the single quotes '.

I ended up the scripts for this particular project on our Linux node and whatever version(s) of Fieldtrip are installed there worked without issue.

What I think I can do--when I find the time--is spin up a couple VMs of Windows 10, 11, Linux, install Matlab and FT and see if I can reproduce (or not) the problem in all environments.

@schoffelen
Copy link
Contributor

I still think it's a string vs char issue.
dataset2files works with persistent variables, and returns the cached output of the previous function call (if the input to subsequent function calls is the same). Somehow matlab does not detect chars and strings to be sufficiently different when dealing with persistent variables.

If I cd into /fileio/private, and do

[a,b,c] = dataset2files("/a/b/c.ds", 'ctf_ds') % double quotes

I get:

a =

"/a/b/c.ds"

b =

1×2 string array

"/a/b/c.ds/c"    "/a/b/c.ds/.res4"

c =

1×2 string array

"/a/b/c.ds/c"    "/a/b/c.ds/.meg4"

If right after that I do:

[a,b,c] = dataset2files('/a/b/c.ds', 'ctf_ds') % single quotes

I get:

a =

"/a/b/c.ds"

b =

1×2 string array

"/a/b/c.ds/c"    "/a/b/c.ds/.res4"

c =

1×2 string array

"/a/b/c.ds/c"    "/a/b/c.ds/.meg4"

But if I do:

clear dataset2files
[a,b,c] = dataset2files('/a/b/c.ds', 'ctf_ds') % single quotes

I get:

a =

'/a/b/c.ds'

b =

'/a/b/c.ds/c.res4'

c =

'/a/b/c.ds/c.meg4'

As desired.

@schoffelen
Copy link
Contributor

Given that it is probably not straightforward to robustify this issue across the codebase, I would vote for an explicit check - and if needed - string2char conversion in ft_checkconfig. What does @robertoostenveld think?

@schoffelen
Copy link
Contributor

Following up on my previous comment, I think that theoretically the best way to enforce a char (instead of string) is to adjust ft_getopt. Practically, this would require updating the c-code, and recompiling the mex-file for all OS's ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants