unicode utf-8 encoding causing issue for xml parser about builder HOT 11 CLOSED

@drchainmail,
"UTF-8 is a variable-width encoding that can represent every character in the Unicode character set. It was designed for backward compatibility with ASCII and to avoid the complications of endianness and byte order marks in UTF-16 and UTF-32." [http://en.wikipedia.org/wiki/UTF-8], so you can (and should, as hinted by the default value of xml encoding, which is, guess, utf-8) use utf-8 to represent Unicode characters.
If your xml parser is failing, it is not receiving an utf-8 encoded stream, do you process the resulting string somehow before parsing it?

from builder.

utf-8 is should be fine for any XML parser, as long as the string is actually encoded as utf-8. Can you provide a simple example where builder is not generating proper utf-8 output (given utf-8 input)?

from builder.

I have a similar issue. In the XML-File all special characters are converted but i need it unchanged for further processing.

$KCODE = 'UTF8'
xml = Builder::Markup.new
xml.instruct!(:xml, :encoding => "UTF-8")
xml.sample("Iñtërnâtiônàl")
xml.target!  

But when i try this out i get a XML-File with:

I#241;t#235;rn#226;ti#244;n#224;l (i removed "&" here otherwise the ascii character conversion is not visible here) instead of Iñtërnâtiônàl

from builder.

I can't reproduce this.

# -*- coding: utf-8 -*-
require 'builder'

$KCODE = 'UTF8'
xml = Builder::XmlMarkup.new
xml.instruct!(:xml, :encoding => "UTF-8")
xml.sample("Iñtërnâtiônàl")
puts xml.target!

Gives:

$ ruby -v xxx.rb 
ruby 2.0.0p195 (2013-05-14 revision 40734) [x86_64-darwin12.3.0]
xxx.rb:4: warning: variable $KCODE is no longer effective; ignored
<?xml version="1.0" encoding="UTF-8"?><sample>Iñtërnâtiônàl</sample>
$ 

What version of Ruby are you using? 2.0 reports that $KCODE is ignored.

from builder.

I use Ruby 1.8.7

from builder.

Even with 1.8.7 I get:

$ ruby -v -rubygems xxx.rb 
ruby 1.8.7 (2012-06-29 patchlevel 370) [i686-darwin12.2.0]
<?xml version="1.0" encoding="UTF-8"?><sample>Iñtërnâtiônàl</sample>
$ 

Any other environmental issues that may effect this (OS, etc)?

from builder.

We use Windows Server and MS IIS as Webserver, Ruby 1.8.7, Rails 3.0.20.

Please take a look at my post at stackoverflow:
http://stackoverflow.com/questions/17067241/ruby-1-8-7-file-encoding-render-to-string-ansi-instead-of-utf-8

I added a few screenshots, maybe the problem is the file-encoding of the generated xml-file.

https://plus.google.com/photos/101165525768874358940/albums/5889322171920232337?authkey=CKaPqczE7tGdag

from builder.

You are using the latest version of builder, right? (3.2.2)

from builder.

ok, actually not, but i updated the builder gem to 3.2.2 and tested it again and no changes. The special chars are still converted into ascii

from builder.

Wait, i got the following output

C:\Appl_Ruby>gem dependency --reverse-dependencies builder
Gem builder-2.1.2
Used by
actionpack-3.0.11 (builder (> 2.1.2))
activemodel-3.0.11 (builder (> 2.1.2))

Gem builder-3.2.2

from builder.

With version 2.1.2 I get the same output as you. It's a version issue. According to the CHANGELOG, you need at least version 2.2.0 to get the behavior you want.

from builder.