Prior of the model

[CatYukino]

Hi,
Thanks for you answer last time. It really helps! I still have a small question here. Is there any prior distribution when combining these operators randomly? I'm not familiar with genetic programming, while I'm doing some work with model selection, which needs complexity of parameters if we use methods like AIC and BIC. However I even do not know how to define parameters in SR... So I wonder what is the prior here and how to change it first, which may be useful in future works.
Thanks! :)

[MilesCranmer]

The genetic algorithm operator weightings (relative probabilities) are defined here

"""
    MutationWeights(;kws...)

This defines how often different mutations occur. These weightings
will be normalized to sum to 1.0 after initialization.
# Arguments
- `mutate_constant::Float64`: How often to mutate a constant.
- `mutate_operator::Float64`: How often to mutate an operator.
- `swap_operands::Float64`: How often to swap the operands of a binary operator.
- `add_node::Float64`: How often to append a node to the tree.
- `insert_node::Float64`: How often to insert a node into the tree.
- `delete_node::Float64`: How often to delete a node from the tree.
- `simplify::Float64`: How often to simplify the tree.
- `randomize::Float64`: How often to create a random tree.
- `do_nothing::Float64`: How often to do nothing.
- `optimize::Float64`: How often to optimize the constants in the tree, as a mutation.
   Note that this is different from `optimizer_probability`, which is
   performed at the end of an iteration for all individuals.
- `form_connection::Float64`: **Only used for `GraphNode`, not regular `Node`**.
   Otherwise, this will automatically be set to 0.0. How often to form a
   connection between two nodes.
- `break_connection::Float64`: **Only used for `GraphNode`, not regular `Node`**.
   Otherwise, this will automatically be set to 0.0. How often to break a
   connection between two nodes.
"""
Base.@kwdef mutable struct MutationWeights
    mutate_constant::Float64 = 0.048
    mutate_operator::Float64 = 0.47
    swap_operands::Float64 = 0.1
    add_node::Float64 = 0.79
    insert_node::Float64 = 5.1
    delete_node::Float64 = 1.7
    simplify::Float64 = 0.0020
    randomize::Float64 = 0.00023
    do_nothing::Float64 = 0.21
    optimize::Float64 = 0.0
    form_connection::Float64 = 0.5
    break_connection::Float64 = 0.1
end

Then there is also the crossover_probability defined here as having 0.066 probability.

After randomly choosing either crossover or mutation, it samples a random mutation from this function here

"""Sample a mutation, given the weightings."""
function sample_mutation(w::MutationWeights)
    weights = convert(Vector, w)
    return StatsBase.sample(v_mutations, StatsBase.Weights(weights))
end

It then calls the mutation function here

        if mutation_choice == :mutate_constant
...

All of the mutation functions are defined in this file, where you can see the individual random number generator calls.

For example, mutate_operator is defined as

"""Randomly convert an operator into another one (binary->binary; unary->unary)"""
function mutate_operator(
    tree::AbstractExpressionNode{T}, options::Options, rng::AbstractRNG=default_rng()
) where {T}
    if !(has_operators(tree))
        return tree
    end
    node = rand(rng, NodeSampler(; tree, filter=t -> t.degree != 0))
    if node.degree == 1
        node.op = rand(rng, 1:(options.nuna))
    else
        node.op = rand(rng, 1:(options.nbin))
    end
    return tree
end

Which means it samples uniformly from the list of operators possible for a given node.

(The "NodeSampler" can be used to sample nodes of an equation satisfying a particular condition. It can also have an arbitrary weighting on nodes too)

[CatYukino]

Thanks for your answer! It really helps!

[MilesCranmer]

complexity of parameters if we use methods like AIC and BIC

AIC and BIC don't work for genetic algorithm searches because they don't take into account the space of expressions. They are designed assuming a fixed functional form.

(Btw, note that you can change the complexity of operators/variables/constants as desired)

[CatYukino]

Got it. So is there any model selection criteria for Symbolic Regression or, more generally, genetic algorithm when there is no exact functional form?

[MilesCranmer]

There’s a lot of different heuristics you can choose, but it’s all a bit arbitrary. The default one in PySR is to look at the drop in log(MSE) as a function of complexity, and locate the largest drop, which often indicates the “best” function. This is basically just an empirical observation that this works okay (and has been used in standard SR problems for a while). So if you want historical precedence, I would go with that.

But if you are looking for something rigorous here in terms of model selection, it sadly just does not exist (and I don’t think it is possible for it to, even if there’s an argument for it). Basically just think about what you actually want out of your model (interpretability, accuracy, speed, etc), and then try to optimise for that using some metric you define as important.

I discuss this a bit at the end of my talk here: https://m.youtube.com/watch?v=fk2r8y5TfNY Basically even the way we define “complexity” is pretty arbitrary! 🙂

[CatYukino]

Thanks for your answer! I really learned a lot!😊