When processing XML files larger than 100MB, using XmlDocument to load the entire file into memory becomes inefficient, leading to high memory consumption and long processing times. In such scenarios, switching to XmlReader and XmlWriter for stream-based processing offers significant performance benefits. However, several pitfalls can arise during implementation.

DOM-Based XML Processing

This approahc loads the entire XML document into memory.

Creating an XML File with DOM:

public void GenerateXmlDocument(string outputPath)
{
    var doc = new XmlDocument();
    var declaration = doc.CreateXmlDeclaration("1.0", "utf-8", null);
    var mainElement = doc.CreateElement("RootElement");
    mainElement.SetAttribute("Owner", "John Doe");
    var childElement = doc.CreateElement("DataEntry");
    mainElement.AppendChild(childElement);

    doc.AppendChild(declaration);
    doc.AppendChild(mainElement);
    doc.Save(outputPath);
}

Reading an XML File with DOM:

public void ParseXmlDocument(string inputPath)
{
    var doc = new XmlDocument();
    doc.Load(inputPath);
    var root = doc.DocumentElement;
    string attributeValue = root.GetAttribute("Owner");
    Console.WriteLine(attributeValue);
}

Stream-Based XML Processing (SAX-style)

This approach reads or writes XML sequentially, which is memory-efficient for large files.

Creating an XML File with XmlWriter:

public void GenerateXmlStream(string outputPath)
{
    using (var buffer = new MemoryStream())
    {
        var config = new XmlWriterSettings();
        // Using Encoding.UTF8 can cause issues; use UTF8Encoding without BOM.
        config.Encoding = new UTF8Encoding(false);

        using (var writer = XmlWriter.Create(buffer, config))
        {
            writer.WriteStartDocument();
            writer.WriteStartElement("RootElement");
            writer.WriteAttributeString("Owner", "Jane Smith");
            writer.WriteStartElement("DataEntry");
            writer.WriteEndElement();
            writer.WriteEndElement();
            writer.WriteEndDocument();
        }

        string xmlContent = Encoding.UTF8.GetString(buffer.ToArray());
        var tempDoc = new XmlDocument();
        tempDoc.LoadXml(xmlContent);
        tempDoc.Save(outputPath);
    }
}

Reading an XML File with XmlReader:

public void ParseXmlStream(string inputPath)
{
    // Using XmlTextReader is necessary to preserve newline characters.
    using (var reader = new XmlTextReader(inputPath))
    {
        reader.WhitespaceHandling = WhitespaceHandling.Significant;
        string currentNode = "";

        while (reader.Read())
        {
            if (reader.NodeType == XmlNodeType.Element)
            {
                currentNode = reader.Name;
                int nestingLevel = reader.Depth;
                bool isEmptyTag = reader.IsEmptyElement;

                for (int i = 0; i < reader.AttributeCount; i++)
                {
                    reader.MoveToAttribute(i);
                    Console.WriteLine($"{reader.Name}: {reader.Value}");
                }
            }
            else if (reader.NodeType == XmlNodeType.EndElement)
            {
                currentNode = reader.Name;
            }
        }
    }
}

Key Issues and Resolutions

1. Incorrect UTF-16 Encoding in Output When using StringBuilder as the output target for XmlWriter, the XML declaration may incorrectly default to UTF-16. Use FileStream, MemoryStream, or another Stream-based target instead.

2. Error Loading XML String with Byte Order Mark (BOM) When creating an XML string via MemoryStream and loading it with XmlDocument.LoadXml, an error may occur if the encoding includes a BOM.

// Problematic:
settings.Encoding = Encoding.UTF8; // Includes BOM
// Solution:
settings.Encoding = new UTF8Encoding(false); // UTF-8 without BOM

The BOM (character 65279) causes the parser to fail. Using UTF8Encoding(false) resolves this.

3. XmlReader Ignoring Line Breaks in Content By default, XmlReader.Create() normalizes newline characters (\r\n) to a single \n, and may treat whitespace differently. To preserve original whitespace and line breaks, use XmlTextReader directly and set WhitespaceHandling.

// This may not preserve original line breaks:
// using (var reader = XmlReader.Create(path, settings))

// This preserves whitespace:
using (var reader = new XmlTextReader(path))
{
    reader.WhitespaceHandling = WhitespaceHandling.Significant;
}

The XmlTextReader constructor does not normalize newlines by default, unlike the factory method XmlReader.Create.

Combining XmlReader for reading large files with XmlDocument or XmlWriter for processing segments is an effective strategy for handling large XML documents.