reviewboard.diffviewer.parser¶

Diff parsing support.

class ParsedDiff(*, parser: BaseDiffParser, uses_commit_ids_as_revisions: bool = False, has_per_file_revisions: bool = True)[source]¶

Bases: object

Parsed information from a diff.

This stores information on the diff as a whole, along with a list of commits made to the diff and a list of files within each.

Extra data can be stored by the parser, which will be made available in DiffSet.extra_data.

This is flexible enough to accommodate a variety of diff formats, including DiffX files.

This class is meant to be used internally and by subclasses of BaseDiffParser.

New in version 4.0.5.

__init__(*, parser: BaseDiffParser, uses_commit_ids_as_revisions: bool = False, has_per_file_revisions: bool = True) → None[source]¶

Initialize the parsed diff information.

Changed in version 7.0.2:

All arguments must now be passed as keyword arguments.
Added the has_per_file_revisions argument.

Parameters:

parser (BaseDiffParser) – The diff parser that parsed this file.
uses_commit_ids_as_revisions (bool, optional) –
Whether commit IDs are used as file revisions.

See ParsedDiff.uses_commit_ids_as_revisions.
has_per_file_revisions (bool, optional) –
Whether the diff has revisions listed per-file.

Most SCM diffs list a source revision for each file that can be directly tied to the change that introduced the file, but some do not.

If set to False, extra_data will be updated with a has_per_file_revisions=False flag set, helping consumers of the diff handle file lookup in an alternate way.

New in version 7.0.2.

parser: BaseDiffParser¶: The diff parser that parsed this file.

extra_data: JSONDict¶

Extra data to store along with the information on the diff.

The contents will be stored directly in DiffSet.extra_data.

changes: list[ParsedDiffChange]¶

The list of changes parsed in this diff.

There should always be at least one.

uses_commit_ids_as_revisions: bool¶

Whether commit IDs are used as file revisions.

A commit ID will be used if an explicit revision isn’t available for a file. For instance, if a parent diff is available, and a file isn’t present in the parent diff, the file will use the parent diff’s parent commit ID as the parent revision.

__annotations__ = {'changes': 'list[ParsedDiffChange]', 'extra_data': 'JSONDict', 'parser': 'BaseDiffParser', 'uses_commit_ids_as_revisions': 'bool'}¶

class ParsedDiffChange(*, parsed_diff: ParsedDiff)[source]¶

Bases: object

Parsed change information from a diff.

This stores information on a change to a tree, consisting of a set of parsed files and extra data to store (in DiffCommit.extra_data.

This will often map to a commit, or just a typical collection of files in a diff. Traditional diffs will have only one of these. DiffX files may have many (but for the moment, only diffs with a single change can be handled when processing these results).

New in version 4.0.5.

commit_id: _BytesProperty¶

The ID of the commit, parsed from the diff.

This may be None.

Type:: bytes

parent_commit_id: _BytesProperty¶

The ID of the primary parent commit, parsed from the diff.

This may be None.

Type:: bytes

__init__(*, parsed_diff: ParsedDiff) → None[source]¶

Initialize the parsed diff information.

Changed in version 7.0.2: All arguments must now be passed as keyword arguments.

Parameters:: parsed_diff (ParsedDiff) – The parent parsed diff information.

extra_data: JSONDict¶

Extra data to store along with the information on the change.

The contents will be stored directly in DiffCommit.extra_data.

files: list[ParsedDiffFile]¶

The list of files parsed for this change.

There should always be at least one.

property parent_parsed_diff: Optional[ParsedDiff][source]¶

The parent diff object.

Type:: ParsedDiff

__annotations__ = {'_parent': 'weakref.ref[ParsedDiff]', 'commit_id': '_BytesProperty', 'extra_data': 'JSONDict', 'files': 'list[ParsedDiffFile]', 'parent_commit_id': '_BytesProperty'}¶

class ParsedDiffFile(*, parser: Optional[BaseDiffParser] = None, parsed_diff_change: Optional[ParsedDiffChange] = None, **kwargs)[source]¶

Bases: object

A parsed file from a diff.

This stores information on a single file represented in a diff, including the contents of that file’s diff, as parsed by DiffParser or one of its subclasses.

Parsers should set the attributes on this based on the contents of the diff, and should add any data found in the diff.

This class is meant to be used internally and by subclasses of BaseDiffParser.

Changed in version 4.0.6: Added old_symlink_target and py:attr:new_symlink_target.

Changed in version 4.0.5: Diff parsers that manually construct instances must pass in parsed_diff_change instead of parser when constructing the object, and must call discard() after construction if the file isn’t wanted in the results.

orig_filename: _BytesProperty¶

The parsed original name of the file.

Type:: bytes

orig_file_details: _RevisionProperty¶

The parsed file details of the original file.

This will usually be a revision.

Type:: bytes or reviewboard.scmtools.core.Revision

modified_filename: _BytesProperty¶

The parsed modified name of the file.

This may be the same as orig_filename.

Type:: bytes

modified_file_details: _RevisionProperty¶

The parsed file details of the modified file.

This will usually be a revision.

Type:: bytes or reviewboard.scmtools.core.Revision

index_header_value: _BytesProperty¶

The parsed value for an Index header.

If present in the diff, this usually contains a filename, but may contain other content as well, depending on the variation of the diff format.

Type:: bytes

old_symlink_target: _BytesProperty¶

The old target for a symlink.

New in version 4.0.6.

Type:: bytes

new_symlink_target: _BytesProperty¶

The new target for a symlink.

New in version 4.0.6.

Type:: bytes

old_unix_mode: _StrProperty¶

The old UNIX mode for the file.

New in version 4.0.6.

Type:: str

new_unix_mode: _StrProperty¶

The new UNIX mode for the file.

New in version 4.0.6.

Type:: str

__init__(*, parser: Optional[BaseDiffParser] = None, parsed_diff_change: Optional[ParsedDiffChange] = None, **kwargs) → None[source]¶

Initialize the parsed file information.

Changed in version 7.0.2:

The parser deprecation and parsed_diff_change requirement have been re-scheduled for Review Board 8.
All arguments must now be passed as keyword arguments.

Changed in version 4.0.5: Added the parsed_diff_change argument (which will be required in Review Board 6.0).

Deprecated the parser argument (which will be removed in Review Board 6.0).

Parameters:

parser (reviewboard.diffviewer.parser.BaseDiffParser, optional) –
The diff parser that parsed this file.

This is deprecated and will be remoed in Review Board 6.0.
parsed_diff_change (ParsedDiffChange, optional) –
The diff change that owns this file.

This will be required in Review Board 6.0.

parser: BaseDiffParser¶: The diff parser that parsed this file.

binary: bool¶: Whether this represents a binary file.

deleted: bool¶: Whether this represents a file that has been deleted.

moved: bool¶

Whether this represents a file that has been moved/renamed.

The file may or may not be modified in the process.

copied: bool¶

Whether this represents a file that has been copied.

The file may or may not be modified in the process.

is_symlink: bool¶: Whether this represents a file that is a symbolic link to another file.

insert_count: int¶: The number of insert (+) lines found in the file.

delete_count: int¶: The number of delete (-) lines found in the file.

skip: bool¶

Whether this file should be skipped by the parser.

If any of the parser methods set this, the file will stop parsing and will be excluded from results.

extra_data: JSONDict¶

Extra data to store along with the information on the file.

The contents will be stored directly in FileDiff.extra_data.

property parent_parsed_diff_change: Optional[ParsedDiffChange][source]¶

The parent change object.

New in version 4.0.5.

Type:: ParsedDiffChange

property data: bytes[source]¶

The data for this diff.

This must be accessed after finalize() has been called.

discard() → None[source]¶

Discard this from the parent change.

This will remove it from the list of files. It’s intended for use when a diff parser is populating the diff but then determines the file is no longer needed.

New in version 4.0.5.

finalize() → None[source]¶

Finalize the parsed diff.

This makes the diff data available to consumers and closes the buffer for writing.

prepend_data(data: bytes) → None[source]¶

Prepend data to the buffer.

Parameters:: data (bytes) – The data to prepend.

append_data(data: bytes) → None[source]¶

Append data to the buffer.

Parameters:: data (bytes) – The data to append.

__annotations__ = {'_data': 'Optional[bytes]', '_data_io': 'io.BytesIO', '_parent': 'Optional[weakref.ref[ParsedDiffChange]]', 'binary': 'bool', 'copied': 'bool', 'delete_count': 'int', 'deleted': 'bool', 'extra_data': 'JSONDict', 'index_header_value': '_BytesProperty', 'insert_count': 'int', 'is_symlink': 'bool', 'modified_file_details': '_RevisionProperty', 'modified_filename': '_BytesProperty', 'moved': 'bool', 'new_symlink_target': '_BytesProperty', 'new_unix_mode': '_StrProperty', 'old_symlink_target': '_BytesProperty', 'old_unix_mode': '_StrProperty', 'orig_file_details': '_RevisionProperty', 'orig_filename': '_BytesProperty', 'parser': 'BaseDiffParser', 'skip': 'bool'}¶

class BaseDiffParser(*, data: bytes, uses_commit_ids_as_revisions: bool = False, **kwargs)[source]¶

Bases: object

Base class for a diff parser.

This is a low-level, basic foundational interface for a diff parser. It performs type checking of the incoming data and a couple of methods for subclasses to implement.

Most SCM implementations will want to either subclass DiffParser or use DiffXParser.

New in version 4.0.5.

has_per_file_revisions: bool = True¶

Whether the diff has revisions listed per-file.

Most SCM diffs list a source revision for each file that can be directly tied to the change that introduced the file, but some (Mercurial) do not.

Diff parsers that don’t support per-file revisions must set this to False so that consumers can alter their file lookup logic.

New in version 7.0.2.

__init__(*, data: bytes, uses_commit_ids_as_revisions: bool = False, **kwargs) → None[source]¶

Initialize the parser.

Changed in version 7.0.2: All arguments must now be passed as keyword arguments.

Parameters:

data (bytes) – The diff content to parse.
uses_commit_ids_as_revisions (bool) –
Whether commit IDs are used as file revisions.

See ParsedDiff.uses_commit_ids_as_revisions.
**kwargs (dict, unused) – Unused keyword arguments.

Raises:

TypeError – The provided data argument was not a bytes type.

data: bytes¶: The diff data being parsed.

uses_commit_ids_as_revisions: bool¶

Whether commit IDs are used as file revisions.

See ParsedDiff.uses_commit_ids_as_revisions.

parse_diff() → ParsedDiff[source]¶

Parse the diff.

This will parse the content of the file, returning a representation of the diff file and its content.

This must be implemented by subclasses.

Returns:

The resulting parsed diff information.

Return type:

ParsedDiff

Raises:

NotImplementedError – This wasn’t implemented by a subclass.
reviewboard.diffviewer.errors.DiffParserError – There was an error parsing part of the diff. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

raw_diff(diffset_or_commit: Union[DiffCommit, DiffSet]) → bytes[source]¶

Return a raw diff as a string.

This takes a DiffSet or DiffCommit and generates a new, single diff file that represents all the changes made. It’s used to regenerate a diff and serve it up for other tools or processes to use.

This must be implemented by subclasses.

Parameters:

diffset_or_commit (reviewboard.diffviewer.models.diffset.DiffSet or :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` :class:`` reviewboard.diffviewer.models.diffcommit.DiffCommit) –

The DiffSet or DiffCommit to render.

If passing in a DiffSet, only the cumulative diff’s file contents will be returned.

If passing in a DiffCommit, only that commit’s file contents will be returned.

Returns:

The diff composed of all the component FileDiffs.

Return type:

bytes

Raises:

NotImplementedError – This wasn’t implemented by a subclass.
TypeError – The provided diffset_or_commit wasn’t of a supported type.

normalize_diff_filename(filename: str) → str[source]¶

Normalize filenames in diffs.

This returns a normalized filename suitable for populating in FileDiff.source_file or FileDiff.dest_file, or for when presenting a filename to the UI.

By default, this strips off any leading slashes, which might occur due to differences in various diffing methods or APIs.

Subclasses can override this to provide additional methods of normalization.

Parameters:: filename (str) – The filename to normalize.
Returns:: The normalized filename.
Return type:: str

__annotations__ = {'data': 'bytes', 'has_per_file_revisions': 'bool', 'uses_commit_ids_as_revisions': 'bool'}¶

class DiffParser(data: bytes, **kwargs)[source]¶

Bases: BaseDiffParser

Parses diff files, allowing subclasses to specialize parsing behavior.

This class provides the base functionality for parsing Unified Diff files. It looks for common information present in many variations of diffs, such as Index: lines, in order to extract files and their modified content from a diff.

Subclasses can extend the parsing behavior to extract additional metadata or handle special representations of changes. They may want to override the following methods:

parse_special_header()
parse_diff_header()
parse_filename_header()
parse_after_headers()
normalize_diff_filename()

INDEX_SEP = b'==================================================================='[source]¶

A separator string below an Index header.

This is commonly found immediately below an Index: header, meant to help locate the beginning of the metadata or changes made to a file.

Its presence and location is not guaranteed.

files: list[ParsedDiffFile]¶: The list of resuting parsed diff files from this parser.

__annotations__ = {'base_commit_id': 'Optional[str]', 'data': 'bytes', 'files': 'list[ParsedDiffFile]', 'has_per_file_revisions': 'bool', 'lines': 'list[bytes]', 'new_commit_id': 'Optional[str]', 'parsed_diff': 'ParsedDiff', 'parsed_diff_change': 'ParsedDiffChange', 'uses_commit_ids_as_revisions': 'bool'}¶

__init__(data: bytes, **kwargs) → None[source]¶

Initialize the parser.

Changed in version 4.0.5: Added **kwargs.

Parameters:

data (bytes) – The diff content to parse.
**kwargs (dict) –
Keyword arguments to pass to the parent class.

New in version 4.0.5.

Raises:

TypeError – The provided data argument was not a bytes type.

base_commit_id: Optional[str]¶: The ID of the commit this change is based on.

new_commit_id: Optional[str]¶: The new commit ID, if available.

lines: list[bytes]¶: The diff content, split into lines.

parsed_diff: ParsedDiff¶: The parsed diff object.

parsed_diff_change: ParsedDiffChange¶: The parsed diff change object.

parse_diff() → ParsedDiff[source]¶

Parse the diff.

Subclasses should override this if working with a diff format that extracts more than one change from a diff.

New in version 4.0.5: Historically, parse() was the main method used to parse a diff. That’s now used exclusively to parse a list of files for the default parsed_diff_change. The old method is around for compatibility, but is no longer called directly outside of this class.

Returns:: The resulting parsed diff information.
Return type:: ParsedDiff
Raises:: reviewboard.diffviewer.errors.DiffParserError – There was an error parsing part of the diff. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

parse() → Sequence[ParsedDiffFile][source]¶

Parse the diff and return a list of files.

This will parse the content of the file, returning any files that were found.

Version Change:: 4.0.5: Historically, this was the main method used to parse a diff. It’s now used exclusively to parse a list of files for the default parsed_diff_change, and parse_diff() is the main method used to parse a diff. This method is around for compatibility, but is no longer called directly outside of this class.

Returns:: The resulting list of files.
Return type:: list of ParsedDiffFile
Raises:: reviewboard.diffviewer.errors.DiffParserError – There was an error parsing part of the diff. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

parse_diff_line(linenum: int, parsed_file: ParsedDiffFile) → int[source]¶

Parse a line of data in a diff.

This will append the line to the parsed file’s data, and if the content represents active changes to a file, its insert/delete counts will be updated to reflect them.

Parameters:

linenum (int) – The 0-based line number.
parsed_file (ParsedDiffFile) – The current parsed diff file info.

Returns:

The next line number to parse.

Return type:

int

parse_change_header(linenum: int) → tuple[int, Optional[ParsedDiffFile]][source]¶

Parse a header before a change to a file.

This will attempt to parse the following information, starting at the specified line in the diff:

Any special file headers (such as Index: lines) through parse_special_header()
A standard Unified Diff file header (through parse_diff_header())
Any content after the header (through parse_after_headers())

If the special or diff headers are able to populate the original and modified filenames and revisions/file details, and none of the methods above mark the file as skipped (by setting ParsedDiffFile.skip), then this will finish by appending all parsed data and returning a parsed file entry.

Subclasses that need to control parsing logic should override one or more of the above methods.

Parameters:

linenum (int) – The line number to begin parsing.

Returns:

A tuple containing the following:

Tuple:

0 (int) – The next line number to parse.
1 (ParsedDiffFile) – The populated ParsedDiffFile instance for this file

Return type:

tuple

Raises:

reviewboard.diffviewer.errors.DiffParserError – There was an error parsing the change header. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

parse_special_header(linenum: int, parsed_file: ParsedDiffFile) → int[source]¶

Parse a special diff header marking the start of a new file’s info.

This attempts to locate an Index: line at the specified line number, which usually indicates the beginning of file’s information in a diff (for Unified Diff variants that support it). By default, this method expects the line to be found at linenum.

If present, the value found immediately after the Index: will be stored in ParsedDiffFile.index_header_value, allowing subclasses to make a determination based on its contents (which may vary between types of diffs, but should include at least a filename.

If the Index: line is not present, this won’t do anything by default.

Subclasses can override this to parse additional information before the standard diff header. They may also set ParsedFileDiff.skip to skip the rest of this file and begin parsing a new entry at the returned line number.

Parameters:

linenum (int) – The line number to begin parsing.
parsed_file (ParsedDiffFile) – The file currently being parsed.

Returns:

The next line number to parse.

Return type:

int

Raises:

reviewboard.diffviewer.errors.DiffParserError – There was an error parsing the special header. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

parse_diff_header(linenum: int, parsed_file: ParsedDiffFile) → int[source]¶

Parse a standard header before changes made to a file.

This attempts to parse the --- (original) and +++ (modified) file lines, which are usually present right before any changes to the file. By default, this method expects the --- line to be found at linenum.

If found, this will populate ParsedDiffFile.orig_filename, ParsedDiffFile.orig_file_details, ParsedDiffFile.modified_filename, and ParsedDiffFile.modified_file_details.

This calls out to parse_filename_header() to help parse the contents immediately after the --- or +++.

Subclasses can override this to parse these lines differently, or to to process the results of these lines (such as converting special filenames to states like “deleted” or “new file”). They may also set ParsedFileDiff.skip to skip the rest of this file and begin parsing a new entry at the returned line number.

Parameters:

linenum (int) – The line number to begin parsing.
parsed_file (ParsedDiffFile) – The file currently being parsed.

Returns:

The next line number to parse.

Return type:

int

Raises:

reviewboard.diffviewer.errors.DiffParserError – There was an error parsing the diff header. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

parse_after_headers(linenum: int, parsed_file: ParsedDiffFile) → int[source]¶

Parse information after a diff header but before diff data.

This attempts to parse the information found after parse_diff_headers() is called, but before gathering any lines that are part of the diff contents. It’s intended for the few diff formats that may place content at this location.

By default, this does nothing.

Subclasses can override this to provide custom parsing of any lines that may exist here. They may also set ParsedFileDiff.skip to skip the rest of this file and begin parsing a new entry at the returned line number.

Parameters:

linenum (int) – The line number to begin parsing.
parsed_file (ParsedDiffFile) – The file currently being parsed.

Returns:

The next line number to parse.

Return type:

int

Raises:

parse_filename_header(s: bytes, linenum: int) → tuple[bytes, bytes][source]¶

Parse the filename found in a diff filename line.

This parses the value after a --- or +++ indicator (or a special variant handled by a subclass), normalizing the filename and any following file details, and returning both for processing and storage.

Often times, the file details will be a revision for the original file, but this is not guaranteed, and is up to the variation of the diff format.

By default, this will assume that a filename and file details are separated by either a single tab, or two or more spaces. If neither are found, this will fail to parse.

This must parse only the provided value, and cannot parse subsequent lines.

Subclasses can override this behavior to parse these lines another way, or to normalize filenames (handling escaping or filenames with spaces as needed by that particular diff variation).

Parameters:

s (bytes) – The value to parse.
linenum (int) – The line number containing the value to parse.

Returns:

A tuple containing:

Tuple:

0 (bytes) – The filename.
1 (bytes) – The additional file information (as bytes).

Return type:

tuple

Raises:

raw_diff(diffset_or_commit: Union[DiffCommit, DiffSet]) → bytes[source]¶

Return a raw diff as a string.

This takes a DiffSet or DiffCommit and generates a new, single diff file that represents all the changes made. It’s used to regenerate a diff and serve it up for other tools or processes to use.

Subclasses can override this to provide any special logic for building the diff.

Parameters:

The DiffSet or DiffCommit to render.

If passing in a DiffSet, only the cumulative diff’s file contents will be returned.

If passing in a DiffCommit, only that commit’s file contents will be returned.

Returns:

The diff composed of all the component FileDiffs.

Return type:

bytes

Raises:

TypeError – The provided diffset_or_commit wasn’t of a supported type.

class DiffXParser(*, data: bytes, uses_commit_ids_as_revisions: bool = False, **kwargs)[source]¶

Bases: BaseDiffParser

Parser for DiffX files.

This will parse files conforming to the DiffX standard, storing the diff content provided in each file section, as well as all the information available in each DiffX section (options, preamble, metadata) as extra_data. This allows the diffs to be re-built on download.

This parser is sufficient for most any DiffX need, but subclasses can be created that augment the stored extra_data for any of the parsed objects.

New in version 4.0.5: This is experimental in 4.0.x, with plans to make it stable for 5.0. The API may change during this time.

__annotations__ = {}¶

parse_diff() → ParsedDiff[source]¶

Parse the diff.

This will parse the content of the DiffX file, returning a representation of the diff file and its content.

Returns:: The resulting parsed diff information.
Return type:: ParsedDiff
Raises:: reviewboard.diffviewer.errors.DiffParserError – There was an error parsing part of the diff. This may be a corrupted diff, or an error in the parsing implementation. Details are in the error message.

raw_diff(diffset_or_commit: Union[DiffCommit, DiffSet]) → bytes[source]¶

Return a raw diff as a string.

This takes a DiffSet or DiffCommit and generates a new, single DiffX file that represents all the changes made, based on the previously-stored DiffX information in extra_data dictionaries. It’s used to regenerate a DiffX and serve it up for other tools or processes to use.

Parameters:

The DiffSet or DiffCommit to render.

If passing in a DiffSet, the full uploaded DiffX file contents will be returned.

If passing in a DiffCommit, a new DiffX representing only that commit’s contents will be returned. This will lack the main preamble or metadata, or any other changes previously in the DiffX file.

Returns:

The resulting DiffX file contents.

Return type:

bytes

Raises:

TypeError – The provided diffset_or_commit value wasn’t of a supported type.