encoding/xml: very low performance in xml parser

[243083df]

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

True

What operating system and processor architecture are you using (go env)?

Windows

What did you do?

I trying to parse large files with SAX with go, and get decadent performance.
I rewrite code in C#, and get maximum performance.

file, err := os.Open(filename)
handle(err)
defer file.Close()
buffer := bufio.NewReaderSize(file, 1024*1024*256) // 33554432
decoder := xml.NewDecoder(buffer)
for {
        t, _ := decoder.Token()
        if t == nil {
            break
        }
        switch se := t.(type) {
        case xml.StartElement:
            if se.Name.Local == "House" {
                house := House{}
                err := decoder.DecodeElement(&house, &se)
                handle(err)
            }
        }
    }

using (XmlReader reader = XmlReader.Create(filename)
            {
                while (reader.Read())
                {
                    switch (reader.NodeType)
                    {
                        case XmlNodeType.Element:
                            if (reader.Name == "House")
                            {
                                //Code
                            }
                            break;
                    }
                }
            }

What did you expect to see?

Mature and fast xml parser in golang.

What did you see instead?

The bottleneck in SAX xml parsing with go is CPU, instead of low HDD io performance.

[ianlancetaylor]

Can you show us a complete, standalone example program? And provide your input file, or at least tell us about it?

What are the actual performance numbers?

Thanks.

[243083df]

File is very big, around 10-50 millions records.
The xml file looks like this:

<?xml version="1.0" encoding="utf-8"?>
<LIST>
<ELEMENT ATTRIBUTE1="" ATTRIBUTE2="" /><ELEMENT ATTRIBUTE1="" ATTRIBUTE2="" />
</LIST>

[a-h]

I was curious, since I use .NET and Go regularly, so I turned this into a reproduction at https://github.com/a-h/sax

On my MacOS machine, I found that the Go version was much slower (around 20 seconds) than the .NET Core 2.0 one (around 3 seconds) for a file with 10 million elements in it laid out as per the example above.

I put the timing output in the README.md. I noted that the CPU was at 100% during Go execution.

I added an SVG of the Go CPU profile output to the repo.

[243083df]

I think, @a-h's benchmark enough to investigate. I used same code.

[gopherbot]

Change https://golang.org/cl/63390 mentions this issue: unicode: speed-up is16/is32

[243083df]

By the way. Is call len(p) in https://github.com/golang/go/blob/master/src/bufio/bufio.go#L196 necessarily?

[mattn]

xml.NewDecoder already create bufio.Reader.

[243083df]

@mattn Yes, but its create bufio.Reader without minimum read value.
bufio.Reader with minimum read value around 64Mb, speed up parsing 2Gb xml from 2m40s to 2m5s(around 20%, but its not stable) on HDD.

[robfordww]

A lot of speed could be gained if the parser was referring to the xml byte slice instead of copying everything. I am working on such an parser for go, based on the rapidxml library in c++

[saleem-mirza]

I am experiencing same issue. Parsing 1 GB XML takes minutes which C# version completes in few seconds ( < 10)