Skip to content

Commit

Permalink
v1.00
Browse files Browse the repository at this point in the history 
Documentation edits and added an 'omitnan' option to the detrend3 function.
  • Loading branch information
@chadagreene
chadagreene committed Mar 11, 2019
1 parent 7454499 commit 5705817
Show file tree
Hide file tree
Showing 54 changed files with 381 additions and 68 deletions.
Binary file modified .DS_Store
View file
Binary file not shown.
3 changes: 3 additions & 0 deletions .gitignore
View file
@@ -0,0 +1,3 @@

.DS_Store
.DS_Store
2 changes: 2 additions & 0 deletions README.md
View file
Expand Up @@ -18,6 +18,8 @@ If that's not correct, find the CDT folder by typing this into the Matlab Comman

```which cdt -all```

If the which hunt still turns up nothing, that suggests the toolbox hasn't been properly installed.

### ...or as individual files and folders:
The files in this GitHub repository may be slightly more up to date than the prepackaged .mltbx toolbox. So if you want to be on the bleeding edge of innovation, get the cdt folder, put it somewhere Matlab can find it, and then right-click on it from within Matlab and select "Add to Path--Selected folder and subfolders."

Expand Down
98 changes: 97 additions & 1 deletion cdt/demos.xml
View file
@@ -1,6 +1,6 @@
<?xml version="1.0" encoding="utf-8"?>
<demos><!--This is an autogenerated file, please do not modify-->
<name>Climate Data Toolbox</name>
<name>The Climate Data Toolbox for Matlab</name>
<type>toolbox</type>
<icon>HelpIcon.DEMOS</icon>
<website/>
Expand Down Expand Up @@ -39,6 +39,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>air_pressure_documentation</source>
<file>doc/html/air_pressure_documentation.html</file>
</demoitem>
<demoitem>
<label>amo_documentation</label>
<type>other</type>
<source>amo_documentation</source>
<file>doc/html/amo_documentation.html</file>
</demoitem>
<demoitem>
<label>anomaly_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -165,6 +171,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>cube2rect_documentation</source>
<file>doc/html/cube2rect_documentation.html</file>
</demoitem>
<demoitem>
<label>daily_insolation_documentation</label>
<type>other</type>
<source>daily_insolation_documentation</source>
<file>doc/html/daily_insolation_documentation.html</file>
</demoitem>
<demoitem>
<label>deseason_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -207,6 +219,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>ekman_documentation</source>
<file>doc/html/ekman_documentation.html</file>
</demoitem>
<demoitem>
<label>ensemble2bnd_documentation</label>
<type>other</type>
<source>ensemble2bnd_documentation</source>
<file>doc/html/ensemble2bnd_documentation.html</file>
</demoitem>
<demoitem>
<label>enso_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -363,18 +381,36 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>local_documentation</source>
<file>doc/html/local_documentation.html</file>
</demoitem>
<demoitem>
<label>mann_kendall_documentation</label>
<type>other</type>
<source>mann_kendall_documentation</source>
<file>doc/html/mann_kendall_documentation.html</file>
</demoitem>
<demoitem>
<label>mask3_documentation</label>
<type>other</type>
<source>mask3_documentation</source>
<file>doc/html/mask3_documentation.html</file>
</demoitem>
<demoitem>
<label>mld_documentation</label>
<type>other</type>
<source>mld_documentation</source>
<file>doc/html/mld_documentation.html</file>
</demoitem>
<demoitem>
<label>monthly_documentation</label>
<type>other</type>
<source>monthly_documentation</source>
<file>doc/html/monthly_documentation.html</file>
</demoitem>
<demoitem>
<label>nao_documentation</label>
<type>other</type>
<source>nao_documentation</source>
<file>doc/html/nao_documentation.html</file>
</demoitem>
<demoitem>
<label>ncdatelim_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -423,6 +459,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>patchsc_documentation</source>
<file>doc/html/patchsc_documentation.html</file>
</demoitem>
<demoitem>
<label>pet_documentation</label>
<type>other</type>
<source>pet_documentation</source>
<file>doc/html/pet_documentation.html</file>
</demoitem>
<demoitem>
<label>plotpsd_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -459,6 +501,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>reof_documentation</source>
<file>doc/html/reof_documentation.html</file>
</demoitem>
<demoitem>
<label>reshapetimeseries_documentation</label>
<type>other</type>
<source>reshapetimeseries_documentation</source>
<file>doc/html/reshapetimeseries_documentation.html</file>
</demoitem>
<demoitem>
<label>rgb_documentation</label>
<type>other</type>
Expand All @@ -471,6 +519,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>rossby_radius_documentation</source>
<file>doc/html/rossby_radius_documentation.html</file>
</demoitem>
<demoitem>
<label>sam_documentation</label>
<type>other</type>
<source>sam_documentation</source>
<file>doc/html/sam_documentation.html</file>
</demoitem>
<demoitem>
<label>scatstat1_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -513,6 +567,18 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>sineval_documentation</source>
<file>doc/html/sineval_documentation.html</file>
</demoitem>
<demoitem>
<label>solar_radiation_documentation</label>
<type>other</type>
<source>solar_radiation_documentation</source>
<file>doc/html/solar_radiation_documentation.html</file>
</demoitem>
<demoitem>
<label>spei_documentation</label>
<type>other</type>
<source>spei_documentation</source>
<file>doc/html/spei_documentation.html</file>
</demoitem>
<demoitem>
<label>spiralplot_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -543,6 +609,12 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>subsubplot_documentation</source>
<file>doc/html/subsubplot_documentation.html</file>
</demoitem>
<demoitem>
<label>sun_angle_documentation</label>
<type>other</type>
<source>sun_angle_documentation</source>
<file>doc/html/sun_angle_documentation.html</file>
</demoitem>
<demoitem>
<label>tile_documentation</label>
<type>other</type>
Expand Down Expand Up @@ -573,12 +645,36 @@ CDT logo illustration by Adam S. Nelsen (http://www.adamnelsen.com)</description
<source>trend_documentation</source>
<file>doc/html/trend_documentation.html</file>
</demoitem>
<demoitem>
<label>ts_normstrap_documentation</label>
<type>other</type>
<source>ts_normstrap_documentation</source>
<file>doc/html/ts_normstrap_documentation.html</file>
</demoitem>
<demoitem>
<label>tutorial_dates_and_times</label>
<type>other</type>
<source>tutorial_dates_and_times</source>
<file>doc/html/tutorial_dates_and_times.html</file>
</demoitem>
<demoitem>
<label>tutorial_mapping</label>
<type>other</type>
<source>tutorial_mapping</source>
<file>doc/html/tutorial_mapping.html</file>
</demoitem>
<demoitem>
<label>tutorial_netcdf</label>
<type>other</type>
<source>tutorial_netcdf</source>
<file>doc/html/tutorial_netcdf.html</file>
</demoitem>
<demoitem>
<label>tutorial_trends</label>
<type>other</type>
<source>tutorial_trends</source>
<file>doc/html/tutorial_trends.html</file>
</demoitem>
<demoitem>
<label>vfill_documentation</label>
<type>other</type>
Expand Down
47 changes: 45 additions & 2 deletions cdt/detrend3.m
View file
Expand Up @@ -6,6 +6,7 @@
%
% Ad = detrend3(A)
% Ad = detrend3(A,t)
% Ad = detrend3(...,'omitnan')
%
%% Description
%
Expand All @@ -15,6 +16,10 @@
% Ad = detrend3(A,t) specifies times t associated with each slice of A. Times
% t do not need to occur at regular intervals in time.
%
% Ad = detrend3(...,'omitnan') applies detrending even in grid cells that contain
% NaN values. If many grid cells contains spurious NaNs, you may find that this
% option is slower than the default.
%
%% Examples
% For examples, type
%
Expand Down Expand Up @@ -44,18 +49,56 @@
end
end

% Does the user want to detrend grid cells that contain spurious NaNs?
tmp = strncmpi(varargin,'omitnan',4);
if any(tmp)
omitnan = true;
varargin = varargin(~tmp);
else
omitnan = false;
end

%% Perform mathematics:

% Center and scale t to improve fit:
t = (t(:)-mean(t))/std(t);
if omitnan
t = (t(:)-mean(t,'omitnan'))/std(t,'omitnan'); % although times really shouldn't be NaNs, but maybe they are, and that's okay.
else
t = (t(:)-mean(t))/std(t);
end

% Reshape, but only deal with the finite values:
mask = all(isfinite(A),3);
if omitnan
mask = sum(isfinite(A),3)>1;
else
mask = all(isfinite(A),3);
end
A = cube2rect(A,mask);

% Detrend A:
Ad = A - [t ones(N,1)]*([t ones(N,1)]\A);


% Deal with spurious NaNs:
if omitnan
% Determine which elements of the y data are finite:
isf = isfinite(A);

% Which columns of data (or grid cells) contain some NaNs, but not all NaNs?
col = find(sum(isf)>1 & sum(isf)<length(t));

% Loop through the columns (each a grid cell) that we have any hope of solving:
for k = 1:length(col)

% For this grid cell, which indices are finite?
ind = isf(:,col(k));
N = sum(ind);

Ad(ind,col(k)) = A(ind,col(k)) - [t(ind) ones(N,1)]*([t(ind) ones(N,1)]\A(ind,col(k)));

end
end

% Unreshape:
Ad = rect2cube(Ad,mask);

Expand Down
76 changes: 74 additions & 2 deletions cdt/doc/detrend3_documentation.m
View file
Expand Up @@ -9,6 +9,7 @@
%
% Ad = detrend3(A)
% Ad = detrend3(A,t)
% Ad = detrend3(...,'omitnan')
%
%% Description
%
Expand All @@ -18,6 +19,10 @@
% |Ad = detrend3(A,t)| specifies times |t| associated with each slice of A.
% Times |t| do not need to occur at regular intervals in time.
%
% |Ad = detrend3(...,'omitnan')| applies detrending even in grid cells that contain
% |NaN| values. If many grid cells contains spurious NaNs, you may find that this
% option is slower than the default.
%
%% Example 1: A 3D gridded dataset
% This sample dataset has a trend of 3.2 units/timestep everywhere:

Expand Down Expand Up @@ -102,7 +107,74 @@
hold on
polyplot(t,sst_dt_1-mean(sst_dt_1),1,'k','linewidth',3)

%% Example 3: The 'omitnan' option
% In some cases, your gridded dataset may have spurious NaNs in them. That is the case
% for the sst dataset we analyzed in the example above. See, for the 802 months
% of SST data, look at how many values are finite:

figure
imagescn(lon,lat,sum(isfinite(sst),3))
cb = colorbar;
ylabel(cb,'number of finite data values')
caxis([700 802]) % sets color axis limits

%%
% In the figure above, take a look at Hudson Bay in Canada. You'll notice
% that not all of the 802 months have valid data there. As many as 100 months
% of data are missing because it can be hard to get a good SST reading in the
% winter months when ice screws up the measurement. Still, you may need to
% remove the long term trend from the available data.
%
% Here's a comparison of detrending the sst dataset without versus with
% the |'omitnan'| option. Start by using the <trend_documentation.html |trend|>
% function, simply to assess the magnitude of the trend:

figure
subplot(1,2,1)
imagescn(lon,lat,trend(sst,12))
cb = colorbar('location','southoutside');
xlabel(cb,'SST trend \circC/yr')
title 'trend without omitnan'
caxis([-0.04 0.04])
cmocean('balance')

subplot(1,2,2)
imagescn(lon,lat,trend(sst,12,'omitnan'))
cb = colorbar('location','southoutside');
xlabel(cb,'SST trend \circC/yr')
title 'trend with omitnan'
caxis([-0.04 0.04])
cmocean('balance')

%%
% In the figure above, note the difference in Hudson Bay, Canada.
%
% Now detrending works the same way. We'll detrend each dataset, and
% then plot the trends of the detrended datasets just as above:

sst_dt = detrend3(sst);
sst_dt_o = detrend3(sst,'omitnan');

figure
subplot(1,2,1)
imagescn(lon,lat,trend(sst_dt,12))
cb = colorbar('location','southoutside');
xlabel(cb,'SST trend \circC/yr')
caxis([-0.04 0.04])
cmocean('balance')
title 'detrended trend without omitnan'

subplot(1,2,2)
imagescn(lon,lat,trend(sst_dt_o,12,'omitnan'))
cb = colorbar('location','southoutside');
xlabel(cb,'SST trend \circC/yr')
caxis([-0.04 0.04])
cmocean('balance')
title 'detrended trend with omitnan'

%%
% A little bit of signal remains in Hudson Bay, likely due to numerical noise.

%% Author Info
% This function is part of the <http://www.github.com/chadagreene/CDT Climate Data Toolbox for Matlab>.
% The function and supporting documentation were written by Chad A. Greene
% of the University of Texas at Austin.
% The function and supporting documentation were written by Chad A. Greene.
8 changes: 5 additions & 3 deletions cdt/doc/ensemble2bnd_documentation.m
View file
Expand Up @@ -175,6 +175,8 @@
set(findall(h.box, 'tag', 'Upper Whisker'), 'linestyle', '-');
set(findall(h.box, 'tag', 'Lower Whisker'), 'linestyle', '-');




%% Author Info
% The |ensemble2bnd| function and its supporting documenation were written
% by <http://kellyakearney.net Kelly A. Kearney> of the University of
% Washington. The |ensemble2bnd| function is part of the Climate
% Data Toolbox for Matlab.

0 comments on commit 5705817

Please sign in to comment.