Skip to content

[#498] Handle utf-8 bom for json parsing #520

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jnunemaker merged 1 commit into from
Apr 21, 2017
Merged

[#498] Handle utf-8 bom for json parsing #520

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion lib/httparty/parser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,10 @@ def xml
MultiXml.parse(body)
end

UTF8_BOM = "\xEF\xBB\xBF".freeze

def json
JSON.parse(body, :quirks_mode => true, :allow_nan => true)
JSON.parse(body.gsub(/^#{UTF8_BOM}/, ''), :quirks_mode => true, :allow_nan => true)
Copy link
Contributor Author

@rsullivan00 rsullivan00 Apr 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this is the right spot for this logic, or if I should be checking encoding somehow before trying to strip the BOM.

I'm open to suggestions, if there is a better way to do this.

end

def csv
Expand Down
5 changes: 5 additions & 0 deletions spec/httparty/parser_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,11 @@ class MyParser < HTTParty::Parser
allow(@parser).to receive_messages(supports_format?: false)
expect(@parser.parse).to_not be_nil
end

it "ignores utf-8 bom" do
allow(@parser).to receive_messages(body: "\xEF\xBB\xBF\{\"hi\":\"yo\"\}")
expect(@parser.parse).to eq({"hi"=>"yo"})
end
end

describe "#supports_format?" do
Expand Down
12 changes: 12 additions & 0 deletions spec/httparty/request_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,12 @@
expect(@request.send(:parse_response, xml)).to eq({'books' => {'book' => {'id' => '1234', 'name' => 'Foo Bar!'}}})
end

it 'should handle utf-8 bom in xml' do
xml = "\xEF\xBB\xBF<books><book><id>1234</id><name>Foo Bar!</name></book></books>"
@request.options[:format] = :xml
expect(@request.send(:parse_response, xml)).to eq({'books' => {'book' => {'id' => '1234', 'name' => 'Foo Bar!'}}})
end

it 'should handle csv automatically' do
csv = ['"id","Name"', '"1234","Foo Bar!"'].join("\n")
@request.options[:format] = :csv
Expand All @@ -380,6 +386,12 @@
expect(@request.send(:parse_response, json)).to eq({'books' => {'book' => {'id' => '1234', 'name' => 'Foo Bar!'}}})
end

it 'should handle utf-8 bom in json' do
json = "\xEF\xBB\xBF{\"books\": {\"book\": {\"name\": \"Foo Bar!\", \"id\": \"1234\"}}}"
@request.options[:format] = :json
expect(@request.send(:parse_response, json)).to eq({'books' => {'book' => {'id' => '1234', 'name' => 'Foo Bar!'}}})
end

it "should include any HTTP headers in the returned response" do
@request.options[:format] = :html
response = stub_response "Content"
Expand Down