Specific functional solutions

[manuel-morales-a]

Dear all,

I hope you are well. I have a question that is related to this older discussion. In there, the loss for the fully factorised solutions worked well, but now I'm looking for a more general approach to impose features in more specific functional forms.

Say we are trying to find an analytical expression of the form $F(x, y) = f_1 (x) \cdot ( (1+y^2) + f_2 (x) \cdot y)$. With PySR, we want to find solutions for $f_1 (x)$ and $f_2(x)$.

Following the logic of the older post, I started to build more tailored loss functions to favour these expressions. However, this approach does not seem to be working. It goes as follows

1. Favour that the tree.degree is 2 and that the operation is *. Penalise heavily otherwise.
2. Favour that tree.l ($f_1 (x)$) is only a function of $x$. Penalise heavily otherwise.
3. Favour that tree.r.degree ($( (1+y^2) + f_2 (x) \cdot y)$) is 2 and the operation is +. Penalise heavily otherwise.
4. I keep going in this nested approach, to then impose that tree.r.l is what I want it to be ($1 + y^2$), tree.r.r is fully factorised and linear in $y$, etc...

In summary, I apply constraints to the children nodes.

I think this approach might not be working because I'm penalising equally the constraints in tree.l and the whole set of nested constraints in tree.r and its children. Is it, in general, better to penalise heavily what happens in the root node, and then less and less what happens in the children nodes?

Also, maybe there's simply a better way to impose these functional forms.

Thanks for any help!

---

If useful for anyone, the loss looks something like this

function L3_loss(tree, dataset::Dataset{T,L}, options)::L where {T,L}
    # Check if expression is factorized:
    penalty_term = L(0)
    
    # Make sure root is degree 2:
    if tree.degree != 2
        penalty_term += L(10000)
    else
        # Make sure operator is *
        if options.operators.binops[tree.op] != *
            penalty_term += L(1000)
        else
            # Split the expression into two subexpressions at the root node:
            f1 = tree.l
            rest = tree.r
            # Check if it's factorised:
            y_in_f1 = any(node -> node.degree==0 && node.constant==false && node.feature==1, f1)
            # check_is_sum encodes the set of nested constraints in the right children node
            good_sum = check_is_sum(rest, options)
            is_factorized = !y_in_f1 && good_sum
            penalty_term += is_factorized ? L(0) : L(100)
        end
    end

    prediction, flag = eval_tree_array(tree, dataset.X, options)
    
    if !flag
        return L(Inf)
    end
    
    return (
        penalty_term 
        + sum((prediction .- dataset.y) .^ 2) / length(prediction)
    )

[manuel-morales-a]

An update, in case it's useful for someone else:

First, for the nominal size of the outputs that I'm working with, a loss of 10000 or 1000 was too harsh. Smaller losses were key, and it actually gave PySR a useful indication on the best way of mutating the expression trees.

Second, penalising less and less as you go down the tree (be very strict with the first multiplication, a bit less strict with the factorisation of $f_1$ and the sum on the right child, etc) seems to be the correct way of proceeding! Recovering a known, relatively easy couple of functions $f_1$ and $f_2$, has been achieved, as several algebraically equivalent expressions are given in the hall of fome.

Now, the challenge will be doing so with more challenging functions :)