Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is KAN particularly not good at fitting the multiplication operations? #212

Open
SimingShan opened this issue May 17, 2024 · 1 comment
Open

Comments

@SimingShan
Copy link

SimingShan commented May 17, 2024

Hi, what an interesting work! However, during my experiments, i found that KAN is particularly not very good at the multiplications, or maybe it is just the way i implement it is not quite right. For example, the most simple problem of x * y, and i set my experiment to be
model = KAN(width=[2,2,1], grid=5, k=3) (which I assume is the same as the paper)
f = lambda x:x[:,[0]] * x[:,[1]]
dataset = create_dataset(f, n_var=2)
then train the model by
# model.train(dataset, opt="LBFGS", steps=200, lamb=0.00);
and the result turns out to be like this
1715975044451
which is quite different with the paper. I have also tried other hyper-parameters, but still didn't work out, which part of my code is wrong?

@KindXiaoming
Copy link
Owner

KindXiaoming commented May 19, 2024

I would count this as a success. (0,0,0) -> linear, (0,0,1) -> quadratic, (0,1,0) -> linear, (0,1,1) -> quadratic. (1,0,0) -> quadratic, (1,1,0) -> linear. Although the quadratic functions are not symmetric, but this is a random gauge degree of freedom. You may try varying random seeds; if you are lucky, you can get symmetry parabola. What do I mean by gauge degree of freedom is that the following equation holds for any $a$:

$2xy = (x+y+a)^2-(x+a)^2-(y+a)^2+a^2$

so the quadratic functions do not have to be symmetric (in the paper, it's just lucky that it's close to $a=0$).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants