2616: post-review cannot post utf-8 in summary or description

karsten********@gmai***** (Google Code) (Is this you? Claim this profile.)
chipx86
chipx86
June 1, 2012
What version are you running?
RBTools VERSION = (0, 3, 4, 'final', 0, True)

What's the URL of the page containing the problem?
post-review --guess-summary --guess-description


What steps will reproduce the problem?
1. post-review --summary ሴ🐳

What is the expected output? What do you see instead?
Expected output: success
seen output: Traceback (most recent call last):
  File "/usr/bin/post-review", line 9, in <module>
    load_entry_point('RBTools==0.3.4', 'console_scripts', 'post-review')()
  File "/usr/lib/python2.7/dist-packages/rbtools/postreview.py", line 4029, in main
    submit_as=options.submit_as)
  File "/usr/lib/python2.7/dist-packages/rbtools/postreview.py", line 3628, in tempt_fate
    options.summary)
  File "/usr/lib/python2.7/dist-packages/rbtools/postreview.py", line 759, in set_review_request_field
    field: value,
  File "/usr/lib/python2.7/dist-packages/rbtools/postreview.py", line 1081, in api_put
    return self.process_json(self.http_put(path, fields))
  File "/usr/lib/python2.7/dist-packages/rbtools/postreview.py", line 1027, in http_put
    data = urllib2.urlopen(r).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1215, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1174, in do_open
    h.request(req.get_method(), req.get_selector(), req.data, headers)
  File "/usr/lib/python2.7/httplib.py", line 958, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 992, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 954, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 812, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 91: ordinal not in range(128)


What operating system are you using? What browser?
Ubuntu Linux 12.04

Please provide any additional information below.
A possible patch is to add the following lines to def http_put():

         try:
+            if isinstance(url, unicode):
+                url = url.encode('utf-8')
+            if isinstance(body, unicode):
+                body = body.encode('utf-8')
             r = HTTPRequest(url, body, headers, method='PUT')
             data = urllib2.urlopen(r).read()
             self.cookie_jar.save(self.cookie_file)
#1 karsten********@gmai***** (Google Code) (Is this you? Claim this profile.)
Actually body does not need to be converted. 
The problem is that body is a <type 'str'>
url is of <type 'unicode>
Somewhere in the urllib2 package something happens like: url + body
To do unicode + str, the str is converted to unicode. However bytes >128 cannot be represented in unicode, hence the exception.
eg: u'www.reviewboard.com' + "\x81" => error
The solution:
str(u'www.reviewboard.com') + "\x81" => 'www.reviewboard.com\x81'


It seems the str() patch is already in place for the http_post function, but not for http_put.
So can you please update http_put similar to http_post?

         try:
-            r = HTTPRequest(url, body, headers, method='PUT')
+            r = HTTPRequest(str(url), body, headers, method='PUT')
             data = urllib2.urlopen(r).read()
             self.cookie_jar.save(self.cookie_file)



#3 karsten********@gmai***** (Google Code) (Is this you? Claim this profile.)
Patch submitted as review 3133 (http://reviews.reviewboard.org/r/3133/)
  • +
    diff --git a/rbtools/postreview.py b/rbtools/postreview.py
    index 81aff404b1a1b9837dc14caf2578bb9030837e74..f6dec17f6623f0924b5d60039a8cce8a4d1a830b 100755
    --- a/rbtools/postreview.py
    +++ b/rbtools/postreview.py
    @@ -727,7 +727,7 @@ class ReviewBoardServer(object):
             }
     
             try:
    -            r = HTTPRequest(url, body, headers, method='PUT')
    +            r = HTTPRequest(str(url), body, headers, method='PUT')
                 data = urllib2.urlopen(r).read()
                 try:
                     self.cookie_jar.save(self.cookie_file)
chipx86
#4 chipx86
Fixed on master. This will be in the next release.
  • +Fixed
  • +Component-RBTools
    +Milestone-RBTools-Release1.0
  • +chipx86