About removal of classical evaluation.

[AlexandreMasta]

I think this is a big step foward to make it in a rush. I´m not saying it is wrong but I would say it could wait a little longer, when the classical evaluation would not bring any real elo to the engine (proved). It´s maintanance were not really "harming" the engine progress. No "nostalgic" feelings about it. It seems this code would not be removed if treated as a normal "simplifying test" as it should. I think it pragmatically failed as a "simplyfication test" by SF´s stardard rules but made by "The Director´s will". (Do you understand that about 3 ELO in this level maybe represents about +15 or even more in other times? This is due to "shrinking gains" in high level of play. Did you consider this?)

Anyway...as I said...a big step foward, maybe it is reasonable, hope it will bring us a stronger engine in the long term. I miss the time the focus of SF Project was +ELO. Let´s hope the best! Good luck and job.

[Dboingue]

I am curious about the implication about the whole information loop in the NNue Master network training, as it might depart from the available blog from SF12 (to my knowledge, i would like to be informed otherwise with link to follow) where it is made clear that the classical evaluation version of SF at that version, was the "orable" trainer for the NN to approximate on some big position set.

I think I have not missed one blog version page and did not see that SF12 blog explanation had been changed, but with SF16, some new language minimally referring to leela's data was used.

My understanding for now, or hypotheses of understanding, are that minimally, it is about using the training games positions.
The least change from SF12 blog description would then have the classical version of SF (assuming the latest SF during training) still produced the training target vector for the master network to fit in a supervised learning mode. (moderate depth classical SF search on the positions to be faithful to SF12 Blog).

There are other ways to use leela's data that might be a structure change in my opinion about the whole information flow in the design of SF evolution (sorry for the word choice is troubling). But it might requires a minimal bit of information other than how fast the implementations have become.. It is a matter of user interpret-ability of the tool. And it might be very easy to do for those who know. And it might not be the prison that good documentation other than source code might have haunted the developer imagination in open source collaborative projects (if that is the resistance). I know chess likes to trade in secrets and expertise auras, but we are talking about programmable things. Someone ought to have the perspective to answer my question.

If using more than the training games databases ground into position database, and keeping the whole game outcome information as part of the SF master network training. . Then what kind of machine learning procedure is used. How are the games outcome integrated.

Could the isolated repository about WDL conversions from one of the SF developers have something to do with it. How?

but the simplest question is, does SF12 blog explanation of the outer flow of information on the network training coming from classical searches still stand.. I am writing here because I would like the op to please provide some links to where such decision might have been discussed. As it does not seem compatible with SF12 blog. and the absence of blog announcements that would modify that.

I am sorry if missed a blog paragraph though. I think source code is not use documentation. So, it might be a bit from exasperation about that possible tendency, that I might sound a bit tense about this issue.. but really i am deeply interested by how training is done, more than how fast the executable performs.. I think documentation might be a bit more transparent about that. it does not require developer inside knowledge to be able to understand such level of reading. It has nothing to do with quantization (well not directly) or feature reduction at input (well not directly). It can be shared perhaps at the same minimal disclosure level as SF12 blog. I would congratulate such effort in documenting something so important for interpretation by serious users and all lichess users for example.

[ChessOverflow]

Although I'm new incomer here but from a chess programmer's point of view, we can't remove the classical eval until at least 90% of the chess positions have been trained. Maybe it would be better if this removal was done in Stockfish 20.

[PGG106]

the "generalization" is not something you have to enable, it's just a property that is always true, chess has so many positions the fact the net works at all means it has to be generalizing.
Whatever positions you may find where hce and nnue disagree on a static eval pov are irrelevant since pure static eval is never used, as long as the net works correctly together with search there's no problem.
And even if you can find positions where the net completely misevals and the search is wrong as a result, keeping hce around is not a fix to any of that, the way to tackle such an issue would be to train stronger nets.

[ChessOverflow]

Keeping hce is the only way to fix the problem, If you have ever used Stockfish-NNUE by @joergoster, you will understand how it does the correct evaluation using classic eval.

POSITION FEN 8/p4p1p/5kp1/1p6/8/3b2P1/PPp2P1P/2R3K1 w - - 0 34

Classical evaluation: -0.87 (white side)

NNUE evaluation:      1.70 (white side)

Final evaluation:     -0.87 (white side)

This formula shows that how SF detect to use strong classical or NNUE.

      Value psq = Value(abs(eg_value(pos.psq_score())));
      bool  largePsq = psq * 16 > (NNUEThreshold1 + pos.non_pawn_material() / 64) * r50;
      bool  classical = largePsq || (psq > PawnValueMg / 4 && !(pos.this_thread()->nodes & 0xB));
      bool strongClassical = pos.non_pawn_material() < 2 * RookValueMg && pos.count<PAWN>() < 2;

      v = classical || strongClassical ? Evaluation<NO_TRACE>(pos).value() : adjusted_NNUE();

When we have StrongClassical, Why use NNUE?

[cj5716]

that is only because classical was forced into lazyEval through rule50count, it has nothing to do with NNUE evaluation being innacurate, since using lazyEval increases speed, we use it in high psq positions.

[cj5716]

If you have any feelings with regards to SF removing classical, you are welcome to fork SF and reintroduce HCE. However, the decision has been made by the maintainer and will not be changed.

[cj5716]

And btw, the strongClassical just detects how much material we have on the board so that we can use the faster HCE evaluation in completely winning positions.

[Dboingue]

https://github.com/official-stockfish/Stockfish/wiki/Stockfish-FAQ#interpretation-of-the-stockfish-evaluation

[ChessOverflow]

If you have any feelings with regards to SF removing classical, you are welcome to fork SF and reintroduce HCE. However, the decision has been made by the maintainer and will not be changed.

First, I have to see if this problem will be solved by structural changes in network training or not.

Recently, I opened a discussion asking what is the difference between a network that is trained with SallowDepth and a network that is trained without a ShallowDepth? But no one has answered it yet.

• ‎Should I train NNUE with or without PruneAtShallowDepth? #4690

@cj5716 Can you help me in this context?

[cj5716]

I am not experienced in NNUE development, sorry. I think asking someone who as trained a neural network/worked on evaluation extensively is more suitable in this regard.