I am curious why the author chose a genetic algorithm rather than standard backprop to distill the evil. Logistic regression seems like a pretty reasonable choice and it’ll be a lot faster than a genetic algorithm. Add an L1 penalty for sparsity.
In the past I’ve tried distilling not just the eval function but the outcome of the search (something like depth 20) in a neural net. It kind of works, but not very well until the net is pretty big. Deepmind did something similar.
I didn’t think of that, but I guess the conclusion as to performance of the model would have remained the same.
Reading over your article again, it seems like you made the same “mistake” as I did. In other words, the evals you’re seeing in the Lichess PGN aren’t the raw outputs of the Stockfish evaluation function, they’re the output of the search, which is a highly nonlinear function of millions of evals. If I recall from their docs, it’s something like depth 18, so your chances of distilling it with 20k parameters is essentially zero.
(I put mistake in quotes here because Deepmind showed that with a server farm of TPUs it is possible to distill the search as well. So it’s not impossible per se.)
But that’s ok! You’ve created a super fast evaluation function. Instead of trying to distill the depth 18 output, it will be much more realistic to distill the depth zero output, the NNUE. If you rerun those FENs in Stockfish you can pretty quickly create an easier dataset.
This is something that's been on my bucket list for a while now, although I'd do it using "normal" programming instead of DL.
i feel like if you can bruteforce every possible board configuration for the next 3 turns and then make the move that leads to more desirable outcomes, that ought to be enough to thrash most amateurs. Probably not good enough to beat somebody who actually understands chess tactics on a deeper level, but I expect most non-masters are just "winging it" and making things up as they go along, so a machine that can think farther ahead than you would win more often than not.
But like I said, this is all just me fantasizing about a project I haven't even started yet so my hypothesis could easily be wrong.
I built a list of simple games in pygame with code and solver generated by AI.
Have implemented 2048, Minesweeper, Sudoku and chess. First three are single player games have made good progress.
I still don't understand UCI interface and have not thought through chess solving. Hope will take another try this weekend
> To be exact every participant has a maximum of 1024 tokens at his disposal to craft the best chess bot they can.
I'd be quite tempted to try linear genetic programming with a variant of a Brainfuck-style UTM for this kind of constraint. Assuming 1024 tokens = 1024 bytes = 1024 instructions, I think there is some degree of performance that is feasible.
This is really interesting to me because hand-coding BF to do something like chess is absolutely beyond me. I wouldn't even know where to begin. A LGP-coded BF program would almost certainly outperform anything I could design with my own hands.
As part of my undergrad work, I used similar principles to the article (steady state genetic algorithms) to create a bot capable of playing Reversi where the fitness function was loosely defined as a set of "weights" on each square across the 8x8 board. These were used as part of the evaluation function in the classic minimax algorithm.
We trained over the course of 5-6 days, and the end generation was capable of beating an intermediate player but got completely destroyed by experts. It was a fun project!
All this I guess comes after laying the ground work, like implementing a bitboard representation or something else, and implementing the logic for being able to tell, whether a move is valid or invalid. That in itself is already a lot of work. iiuc the idea here is "merely" writing the engine, and take everything else as a given.
The game’s implementation itself was furnished with the competition by Sebastian Lague. I completely agree that writing the move logic, validation, etc… is a difficult undertaking especially when it comes to optimisation which is what allows the bots built on top to perform well.
That makes sense for such a competition. Thanks for making this even clearer!
Of course another interesting competition could be to develop _all_ of a chess game and engine and see how low in code people can go. But that would be an entirely different competition.
Is there not an open standard for this? I glanced and didn’t see anything, but seems crazy every chess project would need to define the game.
There are open components for generic purposes, e.g. draw the board given a PGN, show the moves, check if moves are legal and so on, but they aren't suitable for engine internals. There are several tricks with bits to store positions, hashing schemes for transposition tables and the likes that are key to unlock the maximum performance possible, and they are usually tied to the engine implementation.