How to test engine strength

I've seen a few people share their engine tests, and I just wanted to mention the concept of test suites, as this is the most accurate way of testing an engine.

When you test engines, you want to remove as many variables as possible. Many people will test engines with an opening book, which is a mistake. Unless the opening book is 100% neutral (meaning that every line ends with an evaluation of +0.00) then the test will be inaccurate. The easiest way to remove that problem is to not use opening books. However, this means the engines have to take time trying to evaluate e2-e4 or basic opening moves. This means they will have less time to evaluate the positions that really make a difference.

That is where test suites come in. Suites are special opening books that start games in certain positions in order to measure engine strength. This removes the need for traditional opening books, and also allows the engines to put their full power into processing important positions, and avoid wasting time on such basic positions such as e2-e4, etc.. Another important aspect of test suites is that the results are able to be reproduced 100%, using the same hardware. Playing a standard engine match is not, as an engine may choose to open with e2-e4 one time, and then choosing to open with c2-c4 the next time.

If you really want to get an accurate measurement of engine strength, using test suites is the best way to test engines against each other.

Currently, I use the Silver Openings Suite test suite, but I believe Fritz also ships with a test suite that comes pre-installed, called the Nunn test suite (named after its author, GM John Nunn).

Here's a good article that covers the basics, and gives step by step instructions on how to use the Silver Openings Suite.

http://www.chessbase.com/newsdetail.asp?newsid=6147

Just as a side note, I finished test Firebird 1.1 against Rybka 3 using the Silver Openings Suite, with controls of 4+2. Results were +31=51-11, in favor of Firebird 1.1

Tablebases

If you are a serious computer chess user, end-game tablebases are required. The 3-4-5 bases are widely available for download and are (relatively) small at about ~7GB. However, the 6-man bases are much, much larger. As I do not have 1TB of disk space to devote to the full 6-man set, I use a selected set of them. I have found this subset to be very good at covering many end-game scenarios, so I wanted to share them with you. The set I currently use is as follows.

krppkr > kqrpkr > kqqrkr = 10.91gb
krpkrp > kqrkrp > kqrkqr = 12.77gb

kppkpp > kqpkpp > kqpkqp > kqqkqp > kqqkqq =12.15gb

kpppkn,b > kqppkn,b > kqqpkn,b = 11.53gb
knppkr > kqnpkr > kqqnkr = 11.48gb
kbppkr > kqbpkr > kqqbkr = 12.45gb
rrp vs. q > qrr vs. q = 7.2gb


This entire subset requires about 80GB of space, and provides a good amount of coverage for frequently played 6-man scenarios.

You can download the full 6-man set (or whichever portions you would like) from Steinar H. Gunderson's website below.

Please note - Do not use a "download accelerator" or any other program that will download more than 1 file at a time! If you have more than 1 concurrent download, you will be banned. Please use responsibly.

http://tablebase.sesse.net/

Brandon

LesChats & Monferrat

Jordi has released two new versions of his books, LesChats and Monferrat. Very impressive work, as usual. You can download them at Jordi's website.

Back!

I'm back, and slightly sunburned, but a good time was had by all.

Working very quickly to update Uptown with some changes, a number of very strong books were released while I was gone. Looking forward to getting the new version over to Sedat for the next round of the SCCT 2010 tournament.

Vacation

Returning 20 Feb. See you in a week. :)

Uptown 1.2

1.2 for public release. Download link over there.. --->

Please let me know how you like it, and if you have the time I'd appreciate any games you could send me. PGN works best, but I'll take it however you can send it.

Brandon

7!

Sedat's tournament is over, and Uptown 1.0 finished very well, in 7th place out of 195. I'm very happy with that, especially knowing that version 1.0 had some serious problems, especially in B90.

I'm also happy with Uptown 1.0's performance with Deep Shredder 12, which was the secondary engine used in the tournament. It it's now the top Deep Shredder 12 book, currently in 4th places, behind Rybka, Stockfish and Naum 4.1.

I'm hoping to release the latest version of Uptown (currently 1.13) in 1-2 days.

Congratulations to all the participants of this round of SCCT 2010.

Brandon

Uptown Status

It's been some time since I've posted, but I wanted to assure everyone that the latest version of Uptown will be released after the SCCT2010 results. From what I've heard from Sedat, things are looking very good for Uptown.

If you're wondering what I've been up to recently, I was doing research on the possibilities of black winning via B06/B08, as 1. e4 g6 has always been interesting to me. However, after some very interesting results and about 2 weeks of research, I'm officially giving up on that line. I will tell you, however, if you would like to see Rybka 3 give some very wrong evaluations, play a few B06 games. I may release the B06/B08 book that came out of the research at a later date, I'm not sure.

Hopefully SCCT2010 will be done soon and Uptown will be made public.

Thanks for reading.

Brandon