Djblets 6.0 Release Notes¶

Distributed locks¶

We added a basic distributed lock implementation that can be used to help applications avoid stampede issues.

djblets.protect.locks.CacheLock provides the lock, using the cache backend for coordination. It can be used as a context manager or passed to existing cache functions (such as djblets.cache.backend.cache_memoize()).

For example:

from djblets.cache.backend import cache_memoize
from djblets.protect.locks import CacheLock

lock = CacheLock(key='my-lock')

# As a context manager:
with lock:
    if lock.locked():
        # Do something expensive
        ...

# With cache_memoize:
result = cache_memoize(
    'my-key',
    lambda: generate_expensive_state(),
    lock=lock,
)

Generic rate limiting¶

djblets.protect.ratelimit allows callers to rate limit any operation, with rate limits based on users, IPs, or any other criteria required.

This can be used to limit requests to third-party services or Webhook endpoints, limit access to files, limit changes to profiles or edits on objects, or anything else the application requires.

See our guide on Rate Limiting.

Protection for cache key poisoning¶

When constructing a cache key, callers can provide a list of strings (hard-coded or variables) used to construct the cache key. Each component will be escaped and combined safely.

This can help avoid cache key poisoning, making cache keys safe when they involve dynamic input, like identifiers or domains.

The following support this:

• djblets.cache.backend.cache_memoize()

• djblets.cache.backend.cache_memoize_iter()

• djblets.cache.backend.make_cache_key()

• djblets.cache.synchronizer.GenerationSynchronizer

For example:

from djblets.cache.backend import cache_memoize


result = cache_memoize(
    ['user-stats', user.username, 'full'],
    lambda: build_user_stats(),
)

This is opt-in. If your cache key is composed of variables instead of static strings, you should update your code to be a list of strings.

Locked cache operations¶

To avoid stampede issues, where multiple concurrent requests may cause data to be generated multiple times for the same cache key, data generation can now be guarded with a distributed lock.

Both cache_memoize() and cache_memoize_iter() now take an optional lock= argument, which accepts a CacheLock instance. This will make a best attempt to prevent the cache data generation function from being called more than once, blocking other requests until completed.

For example:

from djblets.cache.backend import cache_memoize
from djblets.protect.locks import CacheLock

lock = CacheLock(key='my-lock')

result = cache_memoize(
    'my-key',
    lambda: generate_expensive_state(),
    lock=lock,
)

Simplified creation of columns¶

Datagrid column subclasses no longer need to override __init__(). Instead, they can set attributes directly on the class. This makes columns easier to write, and reduces startup time and memory usage of datagrids.

For example:

from djblets.datagrid.grids import Column


class AuthorColumn(Column):
    label = 'Author'
    detailed_label = 'Author Name'
    db_field = 'author__name'
    shrink = True
    sortable = True

Caller-controlled columns and sorts¶

Datagrids can now be given an explicit list of columns and sorting to use when instantiating the instance.

For example:

datagrid = MyDataGrid(
    request=request,
    columns=['summary', 'author', 'last_updated'],
    sort=['-last_updated'],
)

Control over datagrid search indexing¶

Search engine bots have a tendency to explore too many combinations of column fields and sort options.

Several updates have been made to reduce what a bot can follow:

• Sort and column choices are no longer navigable links.

• The last page of a datagrid now communicates to search engine bots that they shouldn’t navigate there.

• Search engine bots are now informed which page is the first, previous, next, or last pages in the datagrid results.

• Canonical URLs are now provided that exclude extra columns or sort options, reducing how many pages need to be indexed.

• URLs all use sorted arguments, keeping URLs stable for search engines that don’t normalize URLs themselves.

• Search indexing for a datagrid can be turned off by setting allow_search_indexing = False when constructing the datagrid.

For example:

from djblets.datagrid.grids import DataGrid


class MyDataGrid(DataGrid):
    allow_search_indexing = False

It’s still up to the search engine bot to follow any hints set by the datagrid. Many AI bots will ignore these, but most search engines will respect them.

JSON serialization of datagrids¶

Datagrids and their columns can now be serialized to JSON. This can be used to provide a representation of the datagrid that can be returned in an API, or rendered through another means.

To serialize a datagrid to JSON, call DataGrid.to_json().

Columns can provide custom JSON serialization by overriding Column.to_json().

Since there are now two ways to render data (via HTML and JSON), you may want to calculate some common value by overriding the new Column.get_raw_object_value(). By default, the result will be returned by both Column.to_json() and Column.render_data(). If you override either of these methods, you can call your get_raw_object_value() and then use the result however you want.

For example:

from typing import Any

from django.utils.html import format_html
from django.utils.safestring import SafeString
from djblets.datagrid.grids import Column, StatefulColumn
from typelets.django.json import SerializableDjangoJSONValue


class AuthorColumn(Column):
    def get_raw_object_value(
        self,
        state: StatefulColumn,
        obj: Author,
    ) -> dict[str, Any]:
        return {
            'id': obj.pk,
            'name': obj.name,
            'url': obj.get_absolute_url(),
        }

    def to_json(
        self,
        state: StatefulColumn,
        obj: Author,
    ) -> SerializableDjangoJSONValue:
        author_data = self.get_raw_object_value(state, obj)

        return {
            'id': author_data['id'],
            'name': author_data['name'],
            '_links': {
                'profile': {
                    'href': author_data['url'],
                },
           },
        }

    def render_data(
        self,
        state: StatefulColumn,
        obj: Author,
    ) -> SafeString:
        author_data = self.get_raw_object_value(state, obj)

        return format_html(
            '<a data-author-id="{id}" href="{url}">{name}</a>',
            id=author_data['id'],
            name=author_data['name'],
            url=author_data['url'],
        )

Extra data storage for datagrid columns¶

We added a StatefulColumn.extra_data attribute for columns, which they can use to store custom data.

This can be used by the column implementation in any way that it may need.

For example:

from django.db.models import QuerySet
from django.utils.safestring import SafeString, mark_safe
from djblets.datagrid.grids import Column, StatefulColumn


class StarredColumn(Column):
    def augment_queryset(
        self,
        state: StatefulColumn,
        queryset: QuerySet,
    ) -> QuerySet:
        pks = set(
            state.datagrid.request.user.starred_books
           .filter(pk__in=state.datagrid.id_list)
            .values_list('pk', flat=True)
        )

        state.extra_data['starred_ids'] = pks

        return queryset

    def get_raw_object_value(
        self,
        state: StatefulColumn,
        obj: Book,
    ) -> bool:
        return obj.pk in state.extra_data['starred_ids']

    def render_data(
        self,
        state: StatefulColumn,
        obj: Book,
    ) -> SafeString:
        if self.get_raw_object_value(state, obj):
            return mark_safe('⭐️')

        return mark_safe('')

Performance improvements¶

We’ve reduced the work needed to calculate datagrid pagination, making datagrids faster.

Cached value access¶

We added get_object_cached_field().

This returns the value populated for a model’s field when using prefetch_related() or select_related().

If the value is not found in the cache, the value will not be fetched from the database. Instead, typelets.symbols.UNSET() will be returned. This makes it a useful way of safely checking for a cached value quickly without any side effects.

For example:

from djblets.db.query import get_object_cached_field


# These would return the associated values if set.
book = (
    Book.objects
    .select_related('author')
    .prefetch_related('series')
)[0]
author = get_object_cached_field(book, 'author')
series = get_object_cached_field(book, 'series')

# And these would return UNSET, without an additional database query.
book = Book.objects.all()[0]
author = get_object_cached_field(book, 'author')  # UNSET
series = get_object_cached_field(book, 'series')  # UNSET

Typing improvements¶

• Added type hints for get_object_or_none().

  The returned type will always be an instance of the model type that’s passed in.

  For example:

  from typing import reveal_type
  
  from djblets.db.query import get_object_or_none
  
  
  obj = get_object_or_none(Book, pk=123)
  reveal_type(obj)  # This will be Book.
  

Bug fixes¶

• Fixed validation of values in CommaSeparatedValuesField.

Deprecations¶

• Deprecated djblets.db.query_catcher and djblets.db.query_comparator.

  This has been replaced by our new django-assert-queries package. Djblets will include this as a dependency for now, but the dependency will be dropped in a future release. Projects using this functionality should explicitly switch to using and depending on django-assert-queries directly.

Per-app admin modules¶

All .admin modules for an extension are now loaded.

Previously, only the top-level .admin module for an extension would be loaded. Now, all extension-provided apps’ .admin modules will be loaded.

This is supported for all apps defined in Extension.apps.

Improved TemplateHook configuration¶

TemplateHook subclasses can now be customized on the class level.

The following can be set on the class directly, instead of having to pass during construction:

• apply_to

• extra_context

• name

• template_name

Further, they can opt out of rendering on a case-by-case basis by overriding should_render().

For example:

from django.http import HttpRequest
from django.template import Context
from djblets.extensions.hooks import TemplateHook


class MyTemplateHook(TemplateHook):
    name = 'actions'
    apply_to = ['dashboard']
    template_name = 'dashboard/action.html'
    extra_context = {
        'action_title': 'Download CSV',
    }

    def should_render(
        self,
        *,
        request: HttpRequest,
        context: Context,
    ) -> bool:
        return request.user.is_authenticated

Performance and state improvements¶

We reduced the chances for extensions to prematurely reload and cause issues under load.

Extension state is synchronized across servers using the cache backend. This could expire prematurely, causing all extensions to reload. Under high load, this could lead to some temporary errors. State should now last for up to a year in cache, or until extension state has changed.

Concurrency issues could also occur during installation of an extension’s static media. This has been fixed.

Extension customization improvements¶

• Added custom template context for the configure_extension() and extension_list() views.

  This allows applications to specialize these views, adding additional data to include in the templates.

• Full extension bundle IDs can now be passed to the {% ext_css_bundle %} and {% ext_js_bundle %} template tags.

  Full bundle IDs include the extension ID. These can be used instead of the short extension-local bundle ID, in places where that’s more convenient or when rendering a bundle outside of an extension-managed template.

• We fixed various type hint issues in both the Python and JavaScript extension APIs, making it easier to properly write extensions in your IDE.

Easier feature overrides in unit tests¶

override_feature_check and override_feature_checks can now be used as a decorator.

These decorators are used for unit tests that need to simulate enabling or disabling a feature check. It can decorate either classes or functions.

For example:

from djblets.features.testing import (override_feature_check,
                                      override_feature_checks)


@override_feature_check(my_feature_id, enabled=True)
def test_my_feature_1() -> None:
    ...


@override_feature_checks({
    'my_feature_2': True,
    'my_feature_3': False,
})
def test_my_feature_2_3() -> None:
    ...

Dynamic conditions¶

ConditionsField will now reflect dynamically-added condition choices.

Callers should now pass in an instance of a ConditionChoices instead of a class, in order to leverage this.

For example:

from django import forms
from djblets.conditions.choices import ConditionChoices
from djblets.forms.fields import ConditionsField


choices = ConditionChoices([...])


class MyForm(forms.Form):
    conditions = ConditionsField(choices=choices)

Bug fixes¶

• Fixed display issues with ListEditWidget.

  The widget now better fits into the available space where it’s shown, and shows a standard button appearance.

Proxy-aware IP lookup¶

We added djblets.http.requests.get_http_request_ip().

This method returns the requester’s IP, paying attention to the X-Real-IP and X-Forwarded-For headers if available.

Callers using this must be careful to ensure HTTP requests cannot send falsified headers. Web servers must sanitize these headers before they reach Django.

Customization for log timers¶

We added new options, context manager support, and deprecations for log_timed():

• The new logger argument can specify which logger should be used for any log messages.

• The new trace_id argument can specify an ID referenced in an error message, helping people locate logs relevant to the error.

• The new extra argument can be used to specify additional state to pass along with the log message.

• The function can now be used as a context manager, which is a convenience over explicitly calling done() on the log timer.

• All arguments except message must now be passed as keyword-only.

  This will be required starting in Djblets 8.

For example:

import logging

from djblets.log import log_timed


logger = logging.getLogger(__name__)


with log_timed('Doing a thing',
               extra={'request': request},
               logger=logger):
    ...

Bug fixes¶

• Fixed file descriptor leaks when starting or restarting logging.

Enhanced logging¶

• Added enhanced logging for DMARC record fetching.

  When fetching a DMARC record, DEBUG-level messages will be logged to indicate when the fetch begins and ends. If it takes too long, WARNING, ERROR, or CRITICAL log messages will be logged based on the time spent.

• Added enhanced logging for e-mail sending.

  When sending an e-mail using a mail backend, DEBUG-level messages will be logged to indicate when the e-mail is being sent and when delivery has finished. If it takes too long, WARNING, ERROR, or CRITICAL log messages will be logged based on the time spent.

More flexible tool checks¶

Checks for babel, lessc, rollup, and other static media tools are now checked by running npm exec in the list of node module paths, instead of checking for specific files in a specific directory.

Layered settings lookup¶

SiteConfiguration.get() now accepts a layers= argument, which is a list of dictionaries to search for the key.

If the key is present in any of the dictionaries, the value will be returned without checking the stored site configuration or defaults.

This can be used to offer overrides for settings specific to users, teams, feature flag state, etc., all with the convenience of one call.

For example:

from djblets.siteconfig.models import SiteConfiguration


siteconfig = SiteConfiguration.objects.get_current()
team_settings = {
    'public_access': True,
}

# This will prioritize team_settings and fall back to siteconfig and
# then the default.
public_server = siteconfig.get(
    'public_access',
    layers=[team_settings],
    default=False,
)

Enhanced usage of @blocktag¶

• Calls to @blocktag can now specify an explicit tag name.

  For example:

  @blocktag(name='my-block-tag')
  def _my_tag(context, nodelist):
      ...
  

• Template tags implemented using @blocktag now accept keyword arguments.

  For example:

  {% my_block_tag param1=True param2="hi" %}
  

Subnet support for healthcheck IPs¶

settings.DJBLETS_HEALTHCHECK_IPS can now list subnets.

When using HealthCheckView, the configured list of IPs allowed to perform a health check can now include subnets.

For example:

DJBLETS_HEALTHCHECK_IPS = [
    '127.0.0.1',
    '10.0.0.0/8',
]

Define typed variables in templates¶

{% definevar %} can now store non-string variables. Callers can specify the type using as_type. This supports:

• as_type=str: String (default)

• as_type=bool: Boolean (true or 1, case-insensitive)

• as_type=int: Integer (invalid values will raise a TemplateSyntaxError).

For example:

{% load djblets_utils %}

{% definevar "my_count" as_type=int %}123{% enddefinevar %}

{% definevar "my_bool" as_type=bool %}false{% enddefinevar %}

Unique ID generation¶

Added a {% unique_id %} template tag for generating stable unique IDs within a page.

These IDs can be used to dynamically create an ID for an element, making it easy for other attributes to reference that ID. For example:

{% load djblets_utils %}

{% unique_id "my_component" as my_unique_id %}

<div aria-labelledby="{{my_unique_id}}">
 <h2 id="{{my_unique_id}}">Header</h2>
</div>

List serialization customization¶

• Added WebAPIResource.serialize_object_list.

  Subclasses can override this to offer custom serialization of items for a payload, or to offer pre-processing/post-processing of the results.

• Added the serialize_object_list_func argument to WebAPIResponsePaginated.

  Callers can provide a function to offer custom serialization of items for a payload, or to offer pre-processing/post-processing of the results.