2155: post-review fails to upload diff if non-ascii characters are used in diff

jer****@gmai***** (Google Code) (Is this you? Claim this profile.)
chipx86
 chipx86
Sept. 23, 2011
2300 
What version are you running?
RBTools v0.3.2

What steps will reproduce the problem?
1. use post-review to upload the attached patch for a review (note it works in the browser)

Also, I'm aware that this seems to be a problem in Python 2.7's urllib.py (perhaps an Ubuntu only problem, I don't know).

What is the expected output? What do you see instead?
Expected: Patch gets uploaded without a problem using post-review. Actual: See stack trace below.

What operating system are you using? What browser?
I'm on Ubuntu 11.04, Firefox (totally unrelated for this problem)

Here is an example stack trace:
>>> HTTP POSTing to http://review.markusproject.org/api/review-requests/1048/diffs/: {}
Traceback (most recent call last):
  File "/usr/local/bin/post-review", line 9, in <module>
    load_entry_point('RBTools==0.3.2', 'console_scripts', 'post-review')()
  File "/usr/local/lib/python2.7/dist-packages/RBTools-0.3.2-py2.7.egg/rbtools/postreview.py", line 3794, in main
    submit_as=options.submit_as)
  File "/usr/local/lib/python2.7/dist-packages/RBTools-0.3.2-py2.7.egg/rbtools/postreview.py", line 3460, in tempt_fate
    parent_diff_content)
  File "/usr/local/lib/python2.7/dist-packages/RBTools-0.3.2-py2.7.egg/rbtools/postreview.py", line 769, in upload_diff
    fields, files)
  File "/usr/local/lib/python2.7/dist-packages/RBTools-0.3.2-py2.7.egg/rbtools/postreview.py", line 982, in api_post
    return self.process_json(self.http_post(path, fields, files))
  File "/usr/local/lib/python2.7/dist-packages/RBTools-0.3.2-py2.7.egg/rbtools/postreview.py", line 903, in http_post
    data = urllib2.urlopen(r).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1185, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1154, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 955, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 989, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 951, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 809, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 14060: ordinal not in range(128)

Not sure if that will be helpful:

$ locale
LANG=en_CA.UTF-8
LANGUAGE=en_CA:en
LC_CTYPE="en_CA.UTF-8"
LC_NUMERIC="en_CA.UTF-8"
LC_TIME="en_CA.UTF-8"
LC_COLLATE="en_CA.UTF-8"
LC_MONETARY="en_CA.UTF-8"
LC_MESSAGES="en_CA.UTF-8"
LC_PAPER="en_CA.UTF-8"
LC_NAME="en_CA.UTF-8"
LC_ADDRESS="en_CA.UTF-8"
LC_TELEPHONE="en_CA.UTF-8"
LC_MEASUREMENT="en_CA.UTF-8"
LC_IDENTIFICATION="en_CA.UTF-8"
LC_ALL=
sub_tabs.diff 
diff --git a/app/views/layouts/_menu.html.erb b/app/views/layouts/_menu.html.erb
index 20a79ea72e1cfe9f216179f8c56f57ea1f7b9049..caa345a7ee02ca5193a6349bd59fb3dadc65efe6 100755
--- a/app/views/layouts/_menu.html.erb
+++ b/app/views/layouts/_menu.html.erb
@@ -31,15 +31,15 @@
 	       id = "menu_active"
 	     <% end %>>
 
-	<%= link_to I18n.t(:users), :controller => 'students' %> 
+	<%= link_to I18n.t(:users), :controller => 'students' %>
 
         </span>
-        <span <% if 
+        <span <% if
 		  controller.controller_name == 'users'
-	     %>
+	      %>
 	       id = "menu_active"
 	     <% end %>>
-            <%= link_to 'Sections', :controller => 'users' %> 
+            <%= link_to 'Sections', :controller => 'users' %>
         </span>
 
 
diff --git a/app/views/layouts/_sub_menu.html.erb b/app/views/layouts/_sub_menu.html.erb
index dbc14979e301119c9ec23f86551e682ad559bf4d..3dfe03e4ce2f7a7343b8b5606bfb51f8831e4376 100755
--- a/app/views/layouts/_sub_menu.html.erb
+++ b/app/
#1 jer****@gmai***** (Google Code) (Is this you? Claim this profile.) 
BTW. I'm using --diff-filename. Here is the script we use (which in turn uses post-review): https://github.com/MarkUsProject/Markus/blob/master/lib/tools/post-markus-review
#2 Tim.*****@gmai***** (Google Code) (Is this you? Claim this profile.) 
This appears to be caused by a change to the http lib in Python 2.7. There is some discussion on it in this thread - http://groups.google.com/group/reviewboard/browse_thread/thread/a406e73e7c13996f/9e1d7f2c662db181?hl=en&lnk=gst&q=UnicodeDecodeError+with+Visual+Studio+files#9e1d7f2c662db181

I found downgrading to Python 2.6 resolved the issue.
#3 truan*****@kssret******* (Google Code) (Is this you? Claim this profile.) 
Seeing this with RBTools 0.3.3 / latest 'nightly' file and Python 2.7

After reading this issue 11898 on the python site that seemed to be related (http://bugs.python.org/issue11898) I made a change to my local post-review.py and changed line 985 (in http_post method) from 
            r = urllib2.Request(url, body, headers)
to
            r = urllib2.Request(str(url), body, headers)

fixed the issue I was having (BOM in the diff)
#4 jer****@gmai***** (Google Code) (Is this you? Claim this profile.) 
Proposed fix in a review request here: http://reviews.reviewboard.org/r/2568/
chipx86
#5 chipx86
chipx86
#6 chipx86 
Fixed on release-0.3.x (0fff56d)
  • +Fixed
  • +Milestone-RBTools-Release1.0
    +Component-RBTools
  • +chipx86
#7 jer****@gmai***** (Google Code) (Is this you? Claim this profile.) 
Thanks!