I pretty soon saw that It actually wasn’t the sitemap XML, but the XSL (XML stylesheets, browsers use those to make XML readable) that was wrecking havoc. This means that although you couldn’t see them in your browser, the sitemaps themselves were not broken and could still be used by search engines.
Although fiddling with the defer-attribute could solve the issue, I think it’s not a good idea for Autoptimize to try to optimize non-HTML resources as it isn’t WordPress’ core functionality either, so I changed Autoptimize to not act on non-HTML content (i.e. without an HTML-tag or with an xsl:stylesheet-tag). This change is in the upcoming 1.7.3 release, it will be pushed live tomorrow.