Optimisation Issues
Occasionally when fitting a model using ds one can run into optimisation issues. The aim of this page is to give some informal guidance on how to deal with such issues. This is should not be considered a substitute for proper knowledge of how optimisation works! In general such problems can be quite complex so these "quick fixes" may not work. If you come up against problems that are not fixed by these tips, or you feel the results are dubious, please contact me.
I will outline three ways of dealing with optimisation issues below. Before doing that, a brief indication of how to tell when things go wrong...
When fitting a relatively simple detection function model, one might end up with the following errors:
> ds.model <- ds(dat, truncation = 300, formula = ~size, adjustment=NULL)
Fitting half-normal key function
Error in ans.ret[meth, ] <- c(ans$par, ans$value, ans$fevals, ans$gevals, :
number of items to replace is not a multiple of replacement length
All models failed to fit!
Error in ds(dat, truncation = 300, formula = ~size, :
No models could be fitted.
From this it is unclear exactly what has gone wrong.
Using the argument debug.level=3 gives details of the optimisation procedure, so one can investigate what is going on:
> ds.model <- ds(dat, truncation = 300, formula = ~size, adjustment=NULL, debug.level=3)
Fitting half-normal key function
initialvalues= 4.509319 0.005575807
par = 4.509319 0.005575807
par = 4.509319 0.005575807
lt lnl = 1792.573
par = 4.509319 0.005575807
par = 4.509319 0.005575807
lt lnl = 1792.573
par = 4.509319 0.005575807
par = 4.509319 0.005575807
lt lnl = 1792.573
par = 4.509319 0.005575822
par = 4.509319 0.005575822
lt lnl = 1792.573
par = 4.751271 0.975864
par = 4.751271 0.975864
lt lnl = -Inf
par = NaN 0.975864
Error in ans.ret[meth, ] <- c(ans$par, ans$value, ans$fevals, ans$gevals, :
number of items to replace is not a multiple of replacement length
All models failed to fit!
Error in ds(obs.table2, truncation = 300, transect = "line", formula = ~size, :
No models could be fitted.
The last few lines illustrate the issue, the second parameter (the one related to the size covariate) becomes "too big" (0.005575822 to 0.975864) and causes the likelihood to be -Inf, at which point the optimisation fails.
For this kind of error, we have the following three options.
Sometimes this kind of issue is caused by values of the covariate being very big so a rescaling of that covariate is then necessary. Simply scaling by the standard deviation of the covariate can help:
dat$size.scaled <- dat$scale/sd(dat$scale)
ds.model <- ds(dat, truncation = 300, formula = ~size.scaled, adjustment=NULL)
It is important to note that one needs to use the original covariate (size) when computing Horvitz-Thompson estimates of population size if the group size is used in that estimate. i.e. use the unscaled size in the numerator of the estimator.
If this does not work it may be that the optimisation is unduly influenced by the starting values. These are effectively guesses (sophisticated guesses in some cases, but still guesses), so changing them manually may help.
ds.model <- ds(dat, truncation = 300, formula = ~size, adjustment=NULL, initial.values=c(4,-1))
Note that the initial values are set in the order that this printed out in the debug output above.
If neither of the above options work, then another option is to translate the code into an mrds call to ddf.ds. mrds is what actually does the work when you make a call to ds. mrds is much more flexible than Distance but the cost of this is that its model specification is more complex.
One of the options that can be harnessed is using bounds for the parameters. These specify the largest and smallest individual parameters can be. By placing these constraints on the parameters, it is possible to "temper" the optimisation problem, making fitting possible.
An equivalent call is shown below, using the upperbounds argument to specify a maximum for the parameter assocaited with the size coefficient is given below.
mrds.model <- ddf(dsmodel = ~mcds(key = "hn", formula = ~size), method = "ds", meta.data = list(width = 300), control=list(upperbounds=c(Inf,0.3)))
Here we set the first parameter upper bound to be Inf since we are not worried about its value, and the second upper bound (for the size coefficient) to be 0.3.
Note that using mrds is more complicated than fitting the equivalent model in Distance. There are many options, so it is essential to read the documentation before proceeding with the analysis -- it is entirely possible the model you think you are fitting is different from the one you are actually fitting.
The ability to specify bounds will appear in an upcoming release of Distance.