Determining the required variables from Drought 2018 product

[ayushprd]

The ICOS Drought 2018 product has 233 variables, I am opening this issue to decide which variables would we require.

Variables available in Drought 2018

• 'CO2_F_MDS',
• 'CO2_F_MDS_QC',
• 'EBC_CF_METHOD',
• 'EBC_CF_N',
• 'GPP_DT_CUT_05',
• 'GPP_DT_CUT_16',
• 'GPP_DT_CUT_25',
• 'GPP_DT_CUT_50',
• 'GPP_DT_CUT_75',
• 'GPP_DT_CUT_84',
• 'GPP_DT_CUT_95',
• 'GPP_DT_CUT_MEAN',
• 'GPP_DT_CUT_REF',
• 'GPP_DT_CUT_SE',
• 'GPP_DT_CUT_USTAR50',
• 'GPP_DT_VUT_05',
• 'GPP_DT_VUT_16',
• 'GPP_DT_VUT_25',
• 'GPP_DT_VUT_50',
• 'GPP_DT_VUT_75',
• 'GPP_DT_VUT_84',
• 'GPP_DT_VUT_95',
• 'GPP_DT_VUT_MEAN',
• 'GPP_DT_VUT_REF',
• 'GPP_DT_VUT_SE',
• 'GPP_DT_VUT_USTAR50',
• 'GPP_NT_CUT_05',
• 'GPP_NT_CUT_16',
• 'GPP_NT_CUT_25',
• 'GPP_NT_CUT_50',
• 'GPP_NT_CUT_75',
• 'GPP_NT_CUT_84',
• 'GPP_NT_CUT_95',
• 'GPP_NT_CUT_MEAN',
• 'GPP_NT_CUT_REF',
• 'GPP_NT_CUT_SE',
• 'GPP_NT_CUT_USTAR50',
• 'GPP_NT_VUT_05',
• 'GPP_NT_VUT_16',
• 'GPP_NT_VUT_25',
• 'GPP_NT_VUT_50',
• 'GPP_NT_VUT_75',
• 'GPP_NT_VUT_84',
• 'GPP_NT_VUT_95',
• 'GPP_NT_VUT_MEAN',
• 'GPP_NT_VUT_REF',
• 'GPP_NT_VUT_SE',
• 'GPP_NT_VUT_USTAR50',
• 'G_F_MDS',
• 'G_F_MDS_QC',
• 'H_CORR',
• 'H_CORR_25',
• 'H_CORR_75',
• 'H_CORR_JOINTUNC',
• 'H_F_MDS',
• 'H_F_MDS_QC',
• 'H_RANDUNC',
• 'H_RANDUNC_METHOD',
• 'H_RANDUNC_N',
• 'LE_CORR',
• 'LE_CORR_25',
• 'LE_CORR_75',
• 'LE_CORR_JOINTUNC',
• 'LE_F_MDS',
• 'LE_F_MDS_QC',
• 'LE_RANDUNC',
• 'LE_RANDUNC_METHOD',
• 'LE_RANDUNC_N',
• 'LW_IN_ERA',
• 'LW_IN_F',
• 'LW_IN_F_MDS',
• 'LW_IN_F_MDS_QC',
• 'LW_IN_F_QC',
• 'LW_IN_JSB',
• 'LW_IN_JSB_ERA',
• 'LW_IN_JSB_F',
• 'LW_IN_JSB_F_QC',
• 'LW_IN_JSB_QC',
• 'LW_OUT',
• 'NEE_CUT_05',
• 'NEE_CUT_05_QC',
• 'NEE_CUT_16',
• 'NEE_CUT_16_QC',
• 'NEE_CUT_25',
• 'NEE_CUT_25_QC',
• 'NEE_CUT_50',
• 'NEE_CUT_50_QC',
• 'NEE_CUT_75',
• 'NEE_CUT_75_QC',
• 'NEE_CUT_84',
• 'NEE_CUT_84_QC',
• 'NEE_CUT_95',
• 'NEE_CUT_95_QC',
• 'NEE_CUT_MEAN',
• 'NEE_CUT_MEAN_QC',
• 'NEE_CUT_REF',
• 'NEE_CUT_REF_JOINTUNC',
• 'NEE_CUT_REF_QC',
• 'NEE_CUT_REF_RANDUNC',
• 'NEE_CUT_REF_RANDUNC_METHOD',
• 'NEE_CUT_REF_RANDUNC_N',
• 'NEE_CUT_SE',
• 'NEE_CUT_USTAR50',
• 'NEE_CUT_USTAR50_JOINTUNC',
• 'NEE_CUT_USTAR50_QC',
• 'NEE_CUT_USTAR50_RANDUNC',
• 'NEE_CUT_USTAR50_RANDUNC_METHOD',
• 'NEE_CUT_USTAR50_RANDUNC_N',
• 'NEE_VUT_05',
• 'NEE_VUT_05_QC',
• 'NEE_VUT_16',
• 'NEE_VUT_16_QC',
• 'NEE_VUT_25',
• 'NEE_VUT_25_QC',
• 'NEE_VUT_50',
• 'NEE_VUT_50_QC',
• 'NEE_VUT_75',
• 'NEE_VUT_75_QC',
• 'NEE_VUT_84',
• 'NEE_VUT_84_QC',
• 'NEE_VUT_95',
• 'NEE_VUT_95_QC',
• 'NEE_VUT_MEAN',
• 'NEE_VUT_MEAN_QC',
• 'NEE_VUT_REF',
• 'NEE_VUT_REF_JOINTUNC',
• 'NEE_VUT_REF_QC',
• 'NEE_VUT_REF_RANDUNC',
• 'NEE_VUT_REF_RANDUNC_METHOD',
• 'NEE_VUT_REF_RANDUNC_N',
• 'NEE_VUT_SE',
• 'NEE_VUT_USTAR50',
• 'NEE_VUT_USTAR50_JOINTUNC',
• 'NEE_VUT_USTAR50_QC',
• 'NEE_VUT_USTAR50_RANDUNC',
• 'NEE_VUT_USTAR50_RANDUNC_METHOD',
• 'NEE_VUT_USTAR50_RANDUNC_N',
• 'NETRAD',
• 'NIGHT',
• 'P',
• 'PA',
• 'PA_ERA',
• 'PA_F',
• 'PA_F_QC',
• 'PPFD_DIF',
• 'PPFD_IN',
• 'P_ERA',
• 'P_F',
• 'P_F_QC',
• 'RECO_DT_CUT_05',
• 'RECO_DT_CUT_16',
• 'RECO_DT_CUT_25',
• 'RECO_DT_CUT_50',
• 'RECO_DT_CUT_75',
• 'RECO_DT_CUT_84',
• 'RECO_DT_CUT_95',
• 'RECO_DT_CUT_MEAN',
• 'RECO_DT_CUT_REF',
• 'RECO_DT_CUT_SE',
• 'RECO_DT_CUT_USTAR50',
• 'RECO_DT_VUT_05',
• 'RECO_DT_VUT_16',
• 'RECO_DT_VUT_25',
• 'RECO_DT_VUT_50',
• 'RECO_DT_VUT_75',
• 'RECO_DT_VUT_84',
• 'RECO_DT_VUT_95',
• 'RECO_DT_VUT_MEAN',
• 'RECO_DT_VUT_REF',
• 'RECO_DT_VUT_SE',
• 'RECO_DT_VUT_USTAR50',
• 'RECO_NT_CUT_05',
• 'RECO_NT_CUT_16',
• 'RECO_NT_CUT_25',
• 'RECO_NT_CUT_50',
• 'RECO_NT_CUT_75',
• 'RECO_NT_CUT_84',
• 'RECO_NT_CUT_95',
• 'RECO_NT_CUT_MEAN',
• 'RECO_NT_CUT_REF',
• 'RECO_NT_CUT_SE',
• 'RECO_NT_CUT_USTAR50',
• 'RECO_NT_VUT_05',
• 'RECO_NT_VUT_16',
• 'RECO_NT_VUT_25',
• 'RECO_NT_VUT_50',
• 'RECO_NT_VUT_75',
• 'RECO_NT_VUT_84',
• 'RECO_NT_VUT_95',
• 'RECO_NT_VUT_MEAN',
• 'RECO_NT_VUT_REF',
• 'RECO_NT_VUT_SE',
• 'RECO_NT_VUT_USTAR50',
• 'RECO_SR',
• 'RECO_SR_N',
• 'RH',
• 'SWC_F_MDS_1',
• 'SWC_F_MDS_1_QC',
• 'SWC_F_MDS_2',
• 'SWC_F_MDS_2_QC',
• 'SWC_F_MDS_3',
• 'SWC_F_MDS_3_QC',
• 'SWC_F_MDS_4',
• 'SWC_F_MDS_4_QC',
• 'SW_IN_ERA',
• 'SW_IN_F',
• 'SW_IN_F_MDS',
• 'SW_IN_F_MDS_QC',
• 'SW_IN_F_QC',
• 'SW_IN_POT',
• 'SW_OUT',
• 'TA_ERA',
• 'TA_F',
• 'TA_F_MDS',
• 'TA_F_MDS_QC',
• 'TA_F_QC',
• 'TIMESTAMP_END',
• 'TIMESTAMP_START',
• 'TS_F_MDS_1',
• 'TS_F_MDS_1_QC',
• 'TS_F_MDS_2',
• 'TS_F_MDS_2_QC',
• 'USTAR',
• 'VPD_ERA',
• 'VPD_F',
• 'VPD_F_MDS',
• 'VPD_F_MDS_QC',
• 'VPD_F_QC',
• 'WD',
• 'WS',
• 'WS_ERA',
• 'WS_F',
• 'WS_F_QC'

This page has information about the FLUXNET2015 variables. As it is mentioned that the drought 2018 dataset is compatible with the FLUXNET format I think we can use this as a reference.

I compared the variables associated with the existing AMERIFLUX_BASE_HH format and found the following equivalent ones:

Variable	Variable in AMERIFLUX_BASE_HH	Equivalent variable in Drought 2018
wind_direction	WD	WD
UST	USTAR	USTAR
NEE	NEE_PI	Multiple variables
H (sensible heat flux)	H	Multiple variables
LE	LE	Multiple variables
relative_humidity	RH	RH
soilM	SWC_1	Multiple variables
PAR	PPFD_IN	PPFD_IN
surface_downwelling_longwave_flux_in_air	LW_IN	LW_IN_F
specific_humidity	H20	not identified
airT	TA	TA_F
Wspd	WS	WS_F
soilT	TS_1	Multiple variables
air_pressure	PA	PA_F
solar_radiation	SW_IN	SW_IN_F
co2atm	CO2_1	Not identified. CO2_F_MDS?
datetime	TIMESTAMP_START	TIMESTAMP_START
precipitation_flux	P	P_F
TotalResp	RECO_PI	Mutliple variables
FC (Vertical transfer CO2)	FC	Not identified
VPD	VPD	VPD_F

cc: @istfer

[ankurdesai]

Hello, here are some thoughts here.

NEE should match to NEE_VUT_REF (possibly flagged with NEE_VUT_REF_QC if you only want measured and not gap filled data). for random uncertainty use NEE_VUT_REF_RANDUNC, for systematic+random uncertainty use NEE_VUT_REF_JOINTUNC (same pattern for others)

H Sensible heat flux - if you want "energy balance corrected" which removes a known systematic bias (but not entirely correctly) use H_CORR, else use H_F_MDS, same idea for LE

Soil moisture/temperature for sites that have it - take the shallowest level for most applications SWC_F_MDS_1 TS_F_MDS_1

Specific humidity (or water vapor mixing ratio) can be derived from RH. temperature, and air pressure using Clausius-Clapeyron equation (for vapor pressure) and Dalton's law (to convert to mass or volume units)

co2atm is CO2_F_MDS (I believe reported in ppm)

Fluxnet uses two methods to derived respiration (totalResp) and photosynthesis - a nighttime and daytime. If you have to go with one, use nighttime, or use both and treat them as samples with uncertainty. RECO_NT_VUT_REF and uncertainty as RECO_NT_VUT_SE

Correct that FC is not in fluxnet. Ameriflux includes FC because it wants to calculate its own NEE from sum of FC and CO2 profile storage, but they have not made that operational, so we rely on sites where the PI of the site has provided this (NEE_PI) else we take FC (since the second term is usually small). For fluxnet2015 or your drought 2018 ICOS product, that is not an issue. Just use NEE.

Rest look good to me! -ankur desai, pecan flux tower guru

[istfer]

Thanks @ayushprd

The FLUXNET2015 page you found indeed seems to be the proper reference, and it's great that Ankur has also clarified some.

Here are my notes and questions:

• I think you have identified all meteorological variables accurately. You will use the met2CF.csv function in your pipeline, right? Then, we would want airT:TA_F, solar_radiation:SW_IN_F, surface_downwelling_longwave_flux_in_air:LW_IN_F, VPD:VPD_F, air_pressure:PA_F, precipitation_flux:P_F, Wspd:WS_F, wind_direction:WD, relative_humidity:RH, PAR:PPFD_IN,

• Agreed that you don't need to worry about specific_humidity, met2CF.csv calculates it if absent

• I agree with Ankur that co2atm:CO2_F_MDS, soilT:TS_F_MDS_1, soilM:TS_F_MDS_2. Now I got curious about how to make use of the measurements for the other levels for soil temp/moisture but for your case Ayush you don't need to worry about this at the moment.

• For NEE and LE, I think we want the not gap-filled versions, so NEE:NEE_VUT_REF, but is any of these LE ones non-gapfilled? LE_F_MDS, LE_CORR, LE_CORR_25, LE_CORR_75, LE_RANDUNC, LE_RANDUNC_N, LE_CORR_JOINTUNC Doesn't look like it.

• I'm also curious if we should use the uncertainties provided by this dataset or continue using our flux.uncertainty function. In PDA we might move towards fitting observation uncertainty parameters, but for benchmark etc.? Also what about systematic errors? What do you think @ankurdesai and @mdietze?

• I believe in PEcAn we are not utilizing RECO and GPP much, but like Ankur said correct mapping in the format would be TotalResp:RECO_NT_VUT_REF, and GPP:GPP_NT_VUT_REF I think