Update the test requirements and dependencies, i.e. the MEM, WALLTIME, DEPENDENCY and DATA header items.

[robertoostenveld]

Each test script should have these 4 header items. This has partially been done with #2278.

[robertoostenveld]

I observed that they're still test scripts that do not specify any dependencies. The -l option to grep returns files with a match and -L returns files without a match. The wc -l counts the number of lines and hence files.

roboos@mentat001 $ grep -l DEPENDENCY *.m | wc -l
931
roboos@mentat001 $ grep -L DEPENDENCY *.m | wc -l
74

[robertoostenveld]

For finding the dependencies of functions on other functions when refactoring the code, I have been using this in the past (which I dug up from my personal matlab folder on my laptop). It might also be somewhere in fieldtrip/private.

function [outlist, depmat] = mydepfun(inlist, varargin)

% MYDEPFUN creates a dependency matrix
% 
% Use as
%    [outlist, depmat] = mydepfun(inlist)
% where 
%   inlist   = Nx1 cell-array, descibes the rows
%   outlist  = 1xM cell-array, describes the columns
%   depmat   = NxM matrix, see below
% 
% The depmat contains the following values:
%  - 0 if there is no dependency
%  - 1 for the function itself
%  - 2 for a direct dependency

for i=1:length(inlist)
  % we first need only the function name
  [p, f, x] = fileparts(inlist{i});
  inlist{i} = f;
  try
    fprintf('processing "%s"\n', inlist{i});
    dep{i} = depfun(inlist{i}, '-toponly', '-quiet'); % determine the dependencies
    dep{i} = dep{i}(2:end); % the first non-interesting dependency is the function itself
  catch
    fprintf('cannot find "%s"\n', inlist{i});
    dep{i} = {};  % this function cannot be found, so no dependencies
  end
  num(i) = length(dep{i});
  % replace with the full filename, including path
  fullname = which(inlist{i});
  if ~isempty(fullname)
    inlist{i} = fullname;
  else
    inlist{i} = fullfile(p, [f x]);
  end
end

% first make a matrix that will hold all dependencies, including many double ones
depmat = zeros(length(inlist), sum(num));

% mark the dependencies with a value of 2
for i=1:length(inlist)
  offset = sum(num(1:(i-1)));
  depmat(i, (offset+1):(offset+num(i))) = 2;
end
dep = dep(~num==0);
outlist = cat(1, dep{:});

% add the functions from the inlist as dependent on them selves, i.e. value of 1
depmat  = cat(2, eye(length(inlist)), depmat);
outlist = cat(1, inlist(:), outlist(:));

% remove all double occurences
s = unique(outlist);
for i=1:length(s)
  sel = strmatch(s{i}, outlist);
  depmat(:,sel(1)) = max(depmat(:,sel),[], 2);
  outlist(sel(2:end)) = [];
  depmat(:,sel(2:end)) = [];
end

% remove the dependencies on matlab 
sel = zeros(size(outlist));
for i=1:length(outlist)
  sel(i) = ~isempty(strfind(outlist{i}, '/opt/matlab')) | ~isempty(strfind(outlist{i}, '/Application'));
end
depmat  = depmat(:,find(~sel));
outlist = outlist(find(~sel));

[robertoostenveld]

With f41a2f7 I have reduced the memory by 1gb or 2gb for many of the test scripts. I have kept the walltime mostly the same, except that I cleaned it up to have consistent "nice" numbers. There were also still a few scripts with errors in the formatting of the mem and wall time requirements.

The dashboard script now includes a 2gb overhead for the memory, and a 30 minute overhead for the time. I believe that should allow all jobs to finish. It will probably need some tweaking later on.

[contsili]

With #2310 some DATA dependencies were fixed and 3 new functions were added that will help to maintain the DEPENDENCY list up to date.