History

CodeRay Version History

This files lists all changes in the CodeRay library since the 0.8.4 release.

Changes in 1.0

Direct Streaming

CodeRay 1.0 introduces Direct Streaming as a faster and simpler alternative to Tokens. It means that all Scanners, Encoders and Filters had to be rewritten, and that older scanners using the Tokens API are no longer compatible with this version.

The benefit of this change is more speed (benchmarks show 10% to 50% more tokens per second compared to CodeRay 0.9), a simpler API, and less code.

Changes related to the new tokens handling include:
  • CHANGED: The Scanners now call Encoders directly; tokens are not added to a Tokens array, but are send to the Encoder as a method call. The Tokens representation (which can be seen as a cache now) is still present, but as a special case; Tokens just encodes the given tokens into an Array for later use.
  • CHANGED: The token actions (text_token, begin_group etc.) are now public methods of Encoder and Tokens.
  • REWRITE of all Scanners, Encoders, Filters, and Tokens.
  • RENAMED :open and :close actions to :begin_group and :end_group.
  • RENAMED open_token and close_token methods to begin_group and end_group.
  • NEW method #tokens allows to add several tokens to the stream. Tokens and Encoders::Encoder define this method.
  • CHANGED The above name changes also affect the JSON, XML, and YAML encoders. CodeRay 1.0 output will be incompatible with earlier versions.
  • REMOVED TokenStream and the Streamable API and all related features like NotStreamableError are now obsolete and have been removed.

General changes

  • IMPROVED documentation in general; additions, corrections and cleanups
  • FIXED some image links in the documentation
  • IMPROVED Ruby 1.9 support (_._ not in $LOAD_PATH)

Tokens

  • REMOVED methods #stream?, #each_text_token.
  • REMOVED #text and #text_size methods. Use the Text encoder instead.
  • REMOVED special implementation of #each taking a filter parameter. Use TokenKindFilter instead.
  • NEW methods encode_with, count, begin_group, end_group, begin_line, and end_line.

RENAMED: Tokens::AbbreviationForKind

Renamed from ClassOfKind; the term "token class" is no longer used in CodeRay. Instead, tokens have kinds.
See #122.

  • REMOVED token kinds :attribute_name_fat, :attribute_value_fat, :operator_fat, :tag_fat, :xml_text, :open, and :close.
  • ADDED token kinds :filename, :namespace, and :eyecatcher.
  • CHANGED: Don't raise error for unknown token kinds unless in $CODERAY_DEBUG mode.

Encoders::CommentFilter

  • NEW alias :remove_comments

Encoders::Filter

  • NEW option tokens.
  • CHANGED: Now it simply delegates to the output.
  • REMOVED include_text_token? and include_block_token? methods.

Encoders::HTML

The HTML encoder was cleaned up and simplified.

  • CHANGED the default style to :alpha.
  • NEW: HTML 5 and CSS 3 compatible, IE incompatible.
    See #215.
  • ADDED support for :line_number_anchors.
    See #208.
  • CHANGED: Use double click to toggle line numbers in table mode (as single
    click jumps to an anchor.)
  • REMOVED support for :line_numbers => :list.
  • FIXED splitting of lines for :line_numbers => :inline, so that the line
    numbers don't get colored, too.
  • RENAMED Output#numerize to #number, which is an actual English word.

Encoders::LinesOfCode

  • CHANGED: compile and finish methods are now protected.

Encoders::Terminal

  • RENAMED from Encoders::Term, added :term alias.
  • CLEANUP: Use #setup's super, don't use :procedure token class.
  • CHANGED: #token's second parameter is no longer optional.
  • REMOVED colors for obsolete token kinds.
  • FIXED handling of line tokens.

Encoders::Text

  • FIXED default behavior of stripping the trailing newline.

RENAMED: Encoders::TokenKindFilter

Renamed from TokenClassFilter.

  • NEW: Handles token groups.
    See #223.
  • RENAMED include_block_token? to include_group?.

Encoders::Statistic

  • CHANGED: Tokens actions are counted separately.

Encoders::YAML

  • NEW method call for allowing code like CodeRay::Duo[:python => :yaml].(code) in Ruby 1.9.

Scanners::Scanner

  • REMOVED helper method String#to_unix.
  • REMOVED method streamable?.
  • NEW: The #tokenize method also takes an Array of Strings as source. The
    code is highlighted as one and split into parts of the input lengths
    after that using Tokens#split_into_parts.

Scanners::CSS

  • NEW: Rudimentary support for the attr, counter, and counters functions.
    See #224.
  • NEW: Rudimentary support for CSS 3 colors.
  • CHANGED: Attribute selectors are highlighted as :attribute_name instead of :string.

Scanners::Debug

  • NEW: Support for line tokens (begin_line and end_line represented by [ and ].)
  • FIXED: Don't send :error and nil tokens for buggy input any more.
  • FIXED: Closes unclosed tokens at the end of scan_tokens.
  • IMPROVED: Highlight unknown tokens as :error.
  • CHANGED: Raises an error when trying to end an invalid token group.

Scanners::Delphi

  • FIXED: Closes open string groups.

Scanners::Diff

  • NEW: Highlighting of code based on file names.
    See ticket #52.

    Use the :highlight_code option to turn this feature off. It's enabled
    by default.

    This is a very original feature. It enables multi-language highlighting for
    diff files, which is especially helpful for CodeRay development itself. The
    updated version of the scanner test suite generated .debug.diff.html files
    using this.

    Note: This is still experimental. Tokens spanning more than one line
    may get highlighted incorrectly. CodeRay tries to keep scanner states
    between the lines and changes, but the quality of the results depend on
    the scanner.

  • NEW: Inline change highlighting, as suggested by Eric Thomas.
    See ticket #227 for details.

    Use the :inline_diff option to turn this feature off. It's enabled by
    default.

    For single-line changes (that is, a single deleted line followed by a single
    inserted line), this feature surrounds the changed parts with an
    :eyecatcher group which appears in a more saturated background color.
    The implementation is quite complex, and highly experimental. The problem
    with multi-layer tokenizing is that the tokens have to be split into parts.
    If the inline change starts, say, in the middle of a string, then additional
    :end_group and :begin_group tokens must be inserted to keep the group
    nesting intact. The extended Scanner#tokenize method and the new
    Tokens#split_into_parts method take care of this.

  • NEW: Highlight the file name in the change headers as :filename.
  • CHANGED: Highlight unknown lines as :comment instead of :head.

Scanners::HTML

  • FIXED: Closes open string groups.

Scanners::JavaScript

  • IMPROVED: Added NaN and Infinity to list of predefined constants.

Scanners::Java

  • NEW: Package names are highlighted as :namespace.
    See #210.

Scanners::Plaintext

  • IMPROVED: Just returns the string without scanning (faster).

    This is much faster than scanning until /\z/ in Ruby 1.8.

Scanners::Python

  • CHANGED: Docstrings are highlighted as :comment.
    See #190.

NEW: Scanners::Raydebug

Copied from Scanners::Debug, highlights the token dump instead of importing it. It also reacts to the .raydebug file name suffix now.

Scanners::Ruby

  • ADDED more predefined keywords (see http://murfy.de/ruby-constants).
  • IMPROVED support for singleton method definitions.
    See #147.
  • FIXED: Don't highlight methods with a capital letter as constants
    (eg. GL.PushMatrix).
  • NEW: Highlight buggy floats (like .5) as :error.
  • CLEANUP of documentation, names of constants and variables, state handling.
  • NEW: Complicated rule for recognition of foo= style method names.
  • NEW: Handles :keep_state option (a bit; experimental).

    Actually, Ruby checks if there is [~>=], but not => following the name.

Scanners::Scheme

  • CHANGED: Does use :operator instead of :operator_fat now.

Scanners::SQL

  • IMPROVED: Extended list of keywords and functions (thanks to Joshua Galvez).

    See #221.

  • FIXED: Closes open string groups.

Scanners::YAML

  • FIXED: Allow spaces before colon in mappings.

    See #231.

NEW: Styles::Alpha

A style that uses transparent HSLA colors as defined in CSS 3.
See #199.

FileType

  • REMOVED FileType#shebang is a protected method now.
  • NEW: Recognizes .gemspec, .rjs, .rpdf extensions and Capfile as Ruby.

    Thanks to the authors of the TextMate Ruby bundle!

Plugin

  • IMPROVED: register_for sets the plugin_id; it can now be a Symbol.

Internal API changes

  • FIXED Encoders::HTML#token's second parameter is no longer optional.

Changes in 0.9.3

  • FIXED: Documentation of Tokens.
    (#218)

coderay executable

  • NEW: automatic TTY detection (uses Term encoder)
  • NEW: optional 3rd parameter for the filename
  • FIXED: Converted to UNIX format.
  • FIXED: Warn about generated files.
  • FIXED: Ensure line break after the output (especially for LoC counter).

Scanners::JavaScript

  • FIXED: Don't keep state of XML scanner between calls for E4X literals.

Scanners::Java, Scanners::JSON

  • FIXED: Close unfinished strings with the correct token kind.

Changes in 0.9.2

  • NEW Basic tests and a Rakefile are now included in the Gem. [Flameeyes]
    A doc task is also included.
  • FIXED Use $CODERAY_DEBUG for debugging instead of $DEBUG. [Trans]
    (#192)
  • REMOVED Term::Ansicolor was bundled under lib/, but not used. [Flameeyes]
    (#205)
  • WORKAROUND for Ruby bug
    #2745

Encoders::Term

  • FIXED strings are closed correctly
    (#138)
  • FIXED several token kinds had no associated color
    (#139)
  • NEW alias terminal

    NOTE: This encoder will be renamed to Encoders::Terminal in the next release.

Scanners::Debug

  • FIXED Don't close tokens that are not open. Send :error token instead.

Scanners::Groovy

  • FIXED token kind of closing brackets is :operator instead of nil
    (#148)

Scanners::PHP

  • FIXED allow \ operator (namespace separator)
    (#209)

Scanners::YAML

  • FIXED doesn't send debug tokens when $DEBUG is true [Trans]
    (#149)

Changes in 0.9.1

Token classes

  • NEW token classes :complex, :decorator, :imaginary
    (all for Python)
  • REMOVED token class :procedure
    – use :function or :method instead.

Tokens

  • NEW method #scanner

    Stores the scanner.

  • REMOVED methods .write_token, .read_token, .escape, .unescape

    They were only used by the Tokens encoder, which was removed also.

Encoders::Encoder

  • REMOVED Don't require the stringio library.
  • NEW public methods #open_token, #close_token, #begin_line, #end_line
    These methods are called automatically, like #text_token.
  • NEW proteced method #append_encoded_token_to_output

Encoders::Tokens

  • REMOVED – use Tokens#dump and Tokens.load.

Encoders::Filter

  • NEW
    A Filter encoder has another Tokens instance as output.

Encoders::TokenClassFilter

  • NEW

    It takes 2 options, :exclude and :include, that specify which token classes
    to include or exclude for the output. They can be a single token class,
    an Array of classes, or the value :all.

Encoders::CommentFilter

  • NEW

    Removes tokens of the :comment class.

Encoders::LinesOfCode

  • NEW

    Counts the lines of code according to the KINDS_NOT_LOC token class list
    defined by the scanner. It uses the new TokenClassFilter.

    Alias: :loc, as in tokens.loc.

Encoders::JSON

  • NEW

    Outputs tokens in a simple JSON format.

Encoders::Term

  • NEW (beta, by Rob Aldred)

    Outputs code highlighted for a color terminal.

Encoders::HTML

  • NEW option :title (default value is CodeRay output)

    Setting this changes the title of the HTML page.

  • NEW option :highlight_lines (default: nil)

    Highlights the given set of line numbers.
    - REMOVED option :level

    It didn't do anything. CodeRay always outputs XHTML.

Encoders::Text

  • Uses Encoder interface with super and #text_token.

Encoders::XML

  • FIXED (#94)

    It didn't work at all.

Scanners

  • NEW Mapped :h to :c, :cplusplus and :'c++' to :cpp,
    :ecma, :ecmascript, :ecma_script to :java_script,
    :pascal to :delphi, and :plain to :plaintext.

Scanners::Scanner

  • NEW constant KINDS_NOT_LOC

    A list of all token classes not considered in LOC count.
    Added appropriate values for scanners.

  • NEW method #lang returns the scanner's lang, which is its plugin_id.
  • FIXED automatic, safe UTF-8 detection [Ruby 1.9]
  • FIXED column takes care of multibyte encodings [Ruby 1.9]
  • FIXED is dumpable (Tokens store their scanner in an @scanner variable)

Scanners::Cpp

  • NEW (C++)

Scanners::Groovy

  • NEW (beta)

Scanners::Python

  • NEW

Scanners::PHP

  • NEW (based on Stefan Walk's work)

Scanners::SQL

  • NEW (based on code by Josh Goebel)

Scanners::C

  • IMPROVED added a list of :directive tokens that were :reserved before
  • IMPROVED detection of labels
  • IMPROVED allow 1L and 1LL style literals

Scanners::CSS

  • IMPROVED element selectors are highlighted as :type instead of :keyword

Scanners::Delphi

  • IMPROVED Don't cache tokens in CaseIgnoringWordList.

Scanners::Java

  • IMPROVED assert is highlighted as a :keyword now
  • IMPROVED const and goto are highlighted as :reserved
  • IMPROVED false, true, and null are highlighted as :pre_constant
  • IMPROVED threadsafe is no longer a :directive
  • IMPROVED String is highlighted as a :pre_type
  • IMPROVED built-in classes ending with Error or Exception are
    highlighted as a :exception instead of :pre_type

Scanners::JavaScript

  • NEW a list of PREDEFINED_CONSTANTS to be highlighted as :pre_constant
  • NEW XML literals are recognized and highlighted
  • NEW function name highlighting
  • IMPROVED .1 is highlighted a number
  • FIXED strings close with the correct kind when terminated unexpectedly

Scanners::JSON

  • IMPROVED constants (true, false, nil) are highlighted as :value

Scanners::Ruby

  • IMPROVED Patterns::KEYWORDS_EXPECTING_VALUE for more accurate
    value_expected detection
  • IMPROVED handling of \ as a string delimiter
  • IMPROVED handling of unicode strings; automatic switching to unicode
  • IMPROVED highlighting of self.method definitions
  • REMOVED Patterns::FANCY_START_SAVE (obsolete)
  • FIXED encoding issues [Ruby 1.9]
  • FIXED a problem in early Ruby 1.8.6 patch versions with Regexp.escape

Scanners::YAML

  • IMPROVED indentation detection

Styles::Cycnus

  • changed a few colors (exceptions, inline strings, predefined types)

Plugin

  • NEW method #title

    Set and get the plugin's title. Titles can be arbitrary strings.

  • NEW method #helper loads helpers from different plugins

    Use this syntax: helper 'other_plugin/helper_name'

FileType

  • NEW FileType[] takes Pathname instances
  • NEW regonizes .cc, .cpp, .cp, .cxx, .c++, .C, .hh, .hpp, .h++, .cu extensions (C++)

    Thanks to Sander Cox and the TextMate C bundle.

  • NEW regonizes .pas, .dpr extensions (Delphi)
  • NEW regonizes .gvy, .groovy extensions (Groovy)
  • NEW regonizes .php, .php3, .php4, .php5 extensions (PHP)
  • NEW regonizes .py, .py3, .pyw extensions (Python)
  • NEW regonizes .rxml extension (Ruby)
  • NEW regonizes .sql extension (SQL)
  • File types list was sorted alphabetically.

CaseIgnoringWordList

  • FIXED (#97)

    The default value is no longer ignored.

ForRedCloth

  • FIXED for RedCloth versions 4.2.0+ (#119)

Cleanups

  • warnings about character classes [Ruby 1.9]
  • encoding issues [Ruby 1.9]
  • documentation, code

Also available in: HTML TXT