Encoding error while parsing JSON
You know, given how much JSON is loved by the Ruby and other communities, I sometimes think the tools are not quite as professional yet as I would like.
Case in point: An error I debugged today for Travel IQ, which occured occasionally in our API (which is public – become an affiliate now and earn us yourself some money !).
That’s a Rails 2.3.5 app, with the json and json_pure gems at 1.4.6 .
The helpful exception was:
JSON::ParserError: 345: unexpected token at '{"id":1920944,"name":"Homestead Seattle-Bellevue","street":"3700 132nd Ave Se","zip":"98006","city":"Bellevue"
...
The rest of the json fills some pages. Thanks, JSON gem, for this helpful message !
JSONLint also gave me this error:
syntax error, unexpected TINVALID at line 2 Parsing failed
At least it reformats the JSON and gave me the line.
With trial and error, I could narrow it down to the hotel description text:
>> puts "Homestead Seattle \x1A Bellevue" Homestead Seattle Bellevue
Alright, that’s an escaped ASCII character, the decimal value is 26, referenced as “Substitute” – one of the nonprintable ASCII characters.
Here is a very nice post explaining (for a different example, the famous non-breaking-space) different ways of escaping.
Back to my problem:
>> JSON.load("Das Homestead Seattle \032 Bellevue befindet sich in Bellevue, Wash.".to_json)
JSON::ParserError: 705: unexpected token at '"Das Homestead Seattle \u001a Bellevue befindet sich in Bellevue, Wash."'
from /usr/local/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:in `parse'
from /usr/local/lib/ruby/gems/1.8/gems/json-1.4.6/lib/json/common.rb:146:in `parse'
Now that’s just weak. The to_json method produces JSON that can’t be parsed by JSON.parse ? I think the to_json method originates somewhere in Rails, but still…
Oh well, in this case it worked by replacing the to_json call with JSON.dump .
