3425: PySVN, UTF-8 file with svn:keywords set raises exception trying to upload diff.

thom.******@gmai***** (Google Code) (Is this you? Claim this profile.)
david
david
July 3, 2014
What version are you running?
2.0.1 and master (traceback is from master)

What's the URL of the page containing the problem?

/r/new/
and also posting from rbt post.

What steps will reproduce the problem?
1.  Add a SVN repo to reviewboard.
2.  In that repo, commit a file with UTF-8 data ("utf8_file.txt"), and a "$Id $" line
3.  Also propset  svn:keywords "Id" utf8_file.txt
4.  Commit the file.
5.  Make a change to the file.
6. Post for review.

What is the expected output? 

The review is posted.  

What do you see instead?

On /r/new/, user gets no indication there was a failure.   From rbt, they get 
ERROR: Error uploading diff


One or more fields had errors (HTTP 400, API Error 105)

    path: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)

Your review request still exists, but the diff is not attached.
What operating system are you using? What browser?


Please provide any additional information below.

Traceback (most recent call last):
  File "reviewboard/reviewboard/webapi/resources/diff.py", line 292, in create
    request.FILES.get('parent_diff_path'))
  File "reviewboard/reviewboard/reviews/forms.py", line 116, in create
    history)
  File "reviewboard/reviewboard/diffviewer/forms.py", line 69, in create
    request=self.request)
  File "reviewboard/reviewboard/diffviewer/managers.py", line 156, in create_from_upload
    save=save)
  File "reviewboard/reviewboard/diffviewer/managers.py", line 182, in create_from_data
    check_existence=(not parent_diff_file_contents)))
  File "reviewboard/reviewboard/diffviewer/managers.py", line 300, in _process_files
    request=request))):
  File "reviewboard/reviewboard/scmtools/models.py", line 239, in get_file_exists
    base_commit_id, request)
  File "reviewboard/reviewboard/scmtools/models.py", line 434, in _get_file_exists_uncached
    exists = self.get_scmtool().file_exists(path, revision)
  File "reviewboard/reviewboard/scmtools/core.py", line 156, in file_exists
    self.get_file(path, revision)
  File "reviewboard/reviewboard/scmtools/svn/__init__.py", line 117, in get_file
    return self.client.get_file(path, revision)
  File "reviewboard/reviewboard/scmtools/svn/pysvn.py", line 106, in get_file
    return self._do_on_path(self._get_file_data, path, revision)
  File "reviewboard/reviewboard/scmtools/svn/pysvn.py", line 72, in _do_on_path
    return cb(normpath, normrev)
  File "reviewboard/reviewboard/scmtools/svn/pysvn.py", line 100, in _get_file_data
    data = self.collapse_keywords(data, keywords[normpath])
  File "reviewboard/reviewboard/scmtools/svn/base.py", line 118, in collapse_keywords
    repl, data)
  File "rb_2.0/lib/python2.7/re.py", line 151, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
// $Id: utf8-file.txt 1 2014-06-18 19:34:26Z tgerdes $
This file is a utf-8 file with a BOM. The first three bytes of the file are:
0xef, 0xbb, 0xbf
david
#1 david
I've got a fix.
  • +PendingReview
  • +Component-SCMTools
  • +david
david
#2 david
Fixed in release-2.0.x (1646de6). Thanks!
  • -PendingReview
    +Fixed
#3 sanford********@gmai***** (Google Code) (Is this you? Claim this profile.)
I see that's fixed, but I got the same error with a non-ascii character in file name/path.
I'm using ReviewBoard 2.0.6 on Ubuntu 14.04; Subversion 1.8.10, RBTools 0.6.2 and Python 2.7.9 on Windows Server 2012.
david
#4 david
Is it the exact same traceback?
#5 sanford********@gmai***** (Google Code) (Is this you? Claim this profile.)
Hmm... Probably not, but now I can only get it on monday...
#6 sanford********@gmai***** (Google Code) (Is this you? Claim this profile.)
Hi!
This is the traceback. I just committed a file named "C:\testdir\test - cópy.log" and this appeared:


>>> Error data: {u'fields': {u'path': [u"'ascii' codec can't decode byte 0xef in position 30: ordinal not in range(128)"]}, u'stat': u'fail', u'err': {u'msg': u'One or more fields had errors', u'code': 105}}
Traceback (most recent call last):
  File "C:\Python27\Scripts\rbt-script.py", line 9, in <module>
    load_entry_point('RBTools==0.6.2', 'console_scripts', 'rbt')()
  File "C:\Python27\lib\site-packages\rbtools-0.6.2-py2.7.egg\rbtools\commands\main.py", line 134, in main
    command.run_from_argv([RB_MAIN, command_name] + args)
  File "C:\Python27\lib\site-packages\rbtools-0.6.2-py2.7.egg\rbtools\commands\__init__.py", line 416, in run_from_argv
    exit_code = self.main(*args) or 0
  File "C:\Python27\lib\site-packages\rbtools-0.6.2-py2.7.egg\rbtools\commands\post.py", line 784, in main
    submit_as=self.options.submit_as)
  File "C:\Python27\lib\site-packages\rbtools-0.6.2-py2.7.egg\rbtools\commands\post.py", line 551, in post_request
    raise CommandError(u'\n'.join(error_msg))
rbtools.commands.CommandError: Error uploading diff


One or more fields had errors (HTTP 400, API Error 105)

    path: 'ascii' codec can't decode byte 0xef in position 30: ordinal not in range(128)

Your review request still exists, but the diff is not attached.


Thanks for your help!!
david
#7 david
We need the traceback from the server side, not the client.
#8 sanford********@gmai***** (Google Code) (Is this you? Claim this profile.)
Sorry! There it is:


2014-09-15 12:25:21,187 - INFO -  - Using reviewboard.scmtools.svn.pysvn backend for SVN
2014-09-15 12:25:21,295 - ERROR - None - svn - /api/review-requests/3267/diffs/ - Error uploading new diff: 'ascii' codec can't decode byte 0xef in position 30: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/webapi/resources/diff.py", line 292, in create
    request.FILES.get('parent_diff_path'))
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/reviews/forms.py", line 116, in create
    history)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/diffviewer/forms.py", line 69, in create
    request=self.request)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/diffviewer/managers.py", line 156, in create_from_upload
    save=save)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/diffviewer/managers.py", line 182, in create_from_data
    check_existence=(not parent_diff_file_contents)))
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/diffviewer/managers.py", line 300, in _process_files
    request=request))):
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/models.py", line 272, in get_file_exists
    base_commit_id, request)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/models.py", line 473, in _get_file_exists_uncached
    exists = self.get_scmtool().file_exists(path, revision)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/core.py", line 156, in file_exists
    self.get_file(path, revision)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/svn/__init__.py", line 117, in get_file
    return self.client.get_file(path, revision)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/svn/pysvn.py", line 105, in get_file
    return self._do_on_path(self._get_file_data, path, revision)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/svn/pysvn.py", line 77, in _do_on_path
    raise FileNotFoundError(path, revision, detail=exc)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.6-py2.7.egg/reviewboard/scmtools/errors.py", line 71, in __init__
    'revision': revision,
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 30: ordinal not in range(128)
david
#9 david
Thank you! We'll get this fixed for 2.0.8
#10 sanford********@gmai***** (Google Code) (Is this you? Claim this profile.)
Can't wait! Thank you so much!