3370: condense diffs function runs out of memory on larger installs

christia*********@gmai***** (Google Code) (Is this you? Claim this profile.)
chipx86
chipx86
May 25, 2014
What version are you running?
2.0

What's the URL of the page containing the problem?
N/A

What steps will reproduce the problem?
1. Running "rb-site manage /var/www/codereview condensediffs"

What is the expected output? What do you see instead?

The operation fails with a MemoryError:

> rb-site manage /var/www/codereview condensediffs
Processing 152829 diffs for duplicates...

This may take a while. It is safe to continue using Review Board while this is
processing, but it may temporarily run slower.
Traceback (most recent call last):
  File "/usr/local/bin/rb-site", line 9, in <module>
    load_entry_point('ReviewBoard==2.0', 'console_scripts', 'rb-site')()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0-py2.7.egg/reviewboard/cmdline/rbsite.py", line 1733, in main
    command.run()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0-py2.7.egg/reviewboard/cmdline/rbsite.py", line 1662, in run
    site.run_manage_command(args[0], args[1:])
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0-py2.7.egg/reviewboard/cmdline/rbsite.py", line 626, in run_manage_command
    execute_from_command_line([__file__, cmd] + params)
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/core/management/__init__.py", line 399, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/core/management/__init__.py", line 392, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/core/management/base.py", line 242, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/core/management/base.py", line 285, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/core/management/base.py", line 415, in handle
    return self.handle_noargs(**options)
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0-py2.7.egg/reviewboard/diffviewer/management/commands/condensediffs.py", line 29, in handle_noargs
    info = FileDiff.objects.migrate_all()
  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0-py2.7.egg/reviewboard/diffviewer/managers.py", line 44, in migrate_all
    for filediff in filediffs.iterator():
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/db/models/query.py", line 220, in iterator
    for row in compiler.results_iter():
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/db/models/sql/compiler.py", line 713, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/db/models/sql/compiler.py", line 786, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/db/backends/util.py", line 53, in execute
    return self.cursor.execute(sql, params)
  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.5-py2.7.egg/django/db/backends/mysql/base.py", line 124, in execute
    return self.cursor.execute(query, args)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
    self.errorhandler(self, exc, value)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
    raise errorclass, errorvalue
MemoryError


What operating system are you using? What browser?

Ubuntu 12.04 TLS running RB 2.0 on MySql 5.5 backend

Please provide any additional information below.

I upgraded from RB 1.7.25 to 2.0 and got given the advise to run the above command.
#1 christia*********@gmai***** (Google Code) (Is this you? Claim this profile.)
I just migrated the database from MySQL to postgres (on the same machine). I ran the condensediffs command again and this time it completed successfully. The rb-site script was maxing out at around 2.1GB memory usage in this case.

Not sure if the dumpdb/loaddb commands perform some sort of sanitation on the data or if somehow on postgres the behaviour is different.
chipx86
#2 chipx86
We're releasing 2.0.1 this Sunday or Monday, and that should address the memory issues. Or, at least improve upon them.

This should be fixed now on release-2.0.x (84d17a3)
  • +Fixed
  • +Milestone-Release2.0.x
    +Component-RB-Site
  • +chipx86