Regarding DKL input

[utkarshp1161]

Hi Maxim,

With respect to below notebook:
https://github.com/pycroscopy/atomai/blob/master/examples/notebooks/atomai_dkl_plasmons.ipynb

Is there a particular reason you flatten out the patches?

Since the patches have the neighbourhood information will it be better to not flatten it and use a convnet as the feature-extractor?

[ziatdinovmax]

Custom convnets didn't work well with gpytorch implementation of GPs at the time, which was one of the reason I created GPax :)

[utkarshp1161]

I see, thanks. Would you be interested in accepting a pull request in https://github.com/pycroscopy/atomai/blob/master/atomai/nets/gp.py?

I am thinking of writing : class ConvFeatureExtractor(nn.Module):

Functionality:

• takes the flattened vector x of shape (batch_size, l)
• reshapes it to a patch ( x = x.view(-1, 1, square_root(l),square_root(l) )
• apply convolution
• returns final vector with shape -> embed_dim [batch_size as usual, 1st index]

[ziatdinovmax]

I will be happy to review it, but I wonder what is the benefit of doing this through AtomAI when this functionality already exists in GPax?

[utkarshp1161]

I am using the dkl from atomai to deploy on instrument. Few months back I was playing with dkl both in gpax and torch on preacquired data( see issue - ziatdinovmax/gpax#64 ). I found gpax was unable to learn the features (as I could see the acquisition function plot with each iteration of dkl not evolving well. After 100 steps of DKL, the steps it chose was almost random and not with accordance to the scalarizer).

I played with ‘step_size’ parameter in dkl.fit, and learning changed but didnt really understand the intuition of what is going on. Then I switched to atomai and things worked fine.

Side note: I tried also to just follow (https://github.com/ziatdinovmax/gpax/blob/main/examples/gpax_viDKL_plasmons.ipynb) without the hAE wrappers and still couldn't fix it.

Do you have any thoughts on how shall I proceed?

[ziatdinovmax]

Hmm.. interesting. A few things to consider:

• The default NN architectures in DKL are different between AtomAI and GPax, with GPax's NN being much smaller.
• In GPax, we replace NN weights with normal distribution by default. This acts as a regularization against overfitting but may constrain learning ability, especially when combined with smaller and less expressive NNs. You can turn it off by setting nn_prior=False.
• Generally, the two implementations provide similar results for the same setting (data normalization, NN architecture, learning rate).

[utkarshp1161]

That makes much more sense. Thank you, I will try it out.