parse pgn to list

[sugizo]

from_pgn = []

with open(pgn_file) as pgn:
    while True:
        game = chess.pgn.read_game(pgn)

        if game is None:
            break
        else:
            board = game.board()

            for move in game.mainline_moves():
                board.push(move)
                move_stack = board.move_stack
                variation_san = board.root().variation_san(move_stack)

            # Seven Tag Roster
            headers_event = game.headers.get("Event")
            headers_site = game.headers.get("Site")
            headers_date = game.headers.get("Date")
            headers_round = game.headers.get("Round")
            headers_white = game.headers.get("White")
            headers_black = game.headers.get("Black")
            result = board.result(claim_draw = True)

            # Optional tag pairs
            plycount = board.ply()
            fen = game.headers.get("FEN")

            try:
                content = {'event' : headers_event if headers_event and headers_event != '?' else '',
                            'site' : headers_site if headers_site and headers_site != '?' else '',
                            'date' : headers_date if headers_date and headers_date != '?' else '',
                            'round' : headers_round if headers_round and headers_round != '?' else '',
                            'white' : headers_white if headers_white and headers_white != '?' else '',
                            'black' : headers_black if headers_black and headers_black != '?' else '',
                            'result' : result if result and result != '?' else '',
                            'plycount' : plycount if plycount and plycount != '?' else '',
                            'fen' : fen if fen and fen != '?' else '',
                            'moves' : variation_san if variation_san and variation_san != '?' else ''
                          }
            except:
                pass
            else:
                from_pgn.append(content)

objective
convert pgn to another format csv or json

try
use code above, then write it with another python module :

• csv
• json
• pandas
• polars

problem
for around + 50 mb data, is take 2 hours, just to parse pgn to list,
when write to file (*.csv or *.json), it depends of the python modules used (pandas, polars, etc)

question
is there any way to make it faster when parse pgn to list ?

thanks and best regards

[MarkZH]

The simplest thing to do is to change the for move in game.mainline_moves(): loop from

            for move in game.mainline_moves():
                board.push(move)
                move_stack = board.move_stack
                variation_san = board.root().variation_san(move_stack)

to this

            for move in game.mainline_moves():
                board.push(move)
            move_stack = board.move_stack
            variation_san = board.root().variation_san(move_stack)

In my quick tests (with relatively simple PGN files), this made reading a PGN file about 8 times faster: 128 seconds to read 1000 games before the change, 17 seconds after the change. You don't need the move_stack and variation_san data until the entire game has been read, so you don't need to get this data on every loop iteration.