From 9b2710502466667dde1a9d6ce22d952ae8ad4dc7 Mon Sep 17 00:00:00 2001
From: murphy <murphy@rubychan.de>
Date: Mon, 1 Jan 2007 02:58:58 +0000
Subject: Done:

General:
- Declared version 0.7.6.
- Moved WordList, CaseIgnoringWordList, Plugin, PluginHost and FileType
  into CodeRay namespace. CodeRay should be "clean" now, except for the
  String#to_unix helper function.
- Fixed a bit of documentation.
- CodeRay binary: Prepare for streaming switch.

Scanners:
- Added code= alias for string=.
- Added streaming? method: Is this Scanner in streaming mode?
- Enhanced error info a bit.
- Ruby scanner:
  - Highlights Regexp heredocs now. They may be added to Ruby 1.9.
  - Speedups with better support for Ruby 1.9.
  - Change in whitespace handling (faster and cleaner now.)
  - Speed up some operator recognition (saving two string comparisons).
- Declared C and Plaintext Scanners as Streamable.

Tokens:
- Changed Text/Block token recognition (#is_a? ::String for Ruby 1.9 support).
- New method: Tokens#text yields the code string.
- text_size fixed.
- Token kind shortcuts (like r for reserved) are now defined in
  token_classes.rb (instead of encoders/html/classes.rb).

Encoders:
- Debug Scanner added.
- Base encoder class adds to @out when encoding (if @out is set).
- A little Tokens scanner speedup.
- Text encoder uses text_token.
- Statistic encoder counts block tokens.
- Smaller changes in XML and HTML encoders.

Styles:
- cYcnus style defines a debug class now.

Duo:
- scanner and encoder are now methods. Scanner and Encoder are created (and
  cached) when needed, not earlier.
- Documented.

Tests:
- Disabled encoder and scanner list check (breaks too often).
- Added identity test, which checks if tokens#text matches the input.
- Added nocolor switch.

Developer tools:
- Benchmark uses Encoder#file_extension for output now.
- Rakefile: Support for 19, 18, yarv and ruby switches for easy comparing
  different Ruby versions.
- Statistic: Demos are no longer tests.
---
 lib/coderay/scanner.rb | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

(limited to 'lib/coderay/scanner.rb')

diff --git a/lib/coderay/scanner.rb b/lib/coderay/scanner.rb
index 62327c7..5993b4c 100644
--- a/lib/coderay/scanner.rb
+++ b/lib/coderay/scanner.rb
@@ -117,9 +117,6 @@ module CodeRay
         setup
       end
 
-      # More mnemonic accessor name for the input string.
-      alias code string
-
       def reset
         super
         reset_instance
@@ -131,6 +128,10 @@ module CodeRay
         reset_instance
       end
 
+      # More mnemonic accessor name for the input string.
+      alias code string
+      alias code= string=
+
       # Scans the code and returns all tokens in a Tokens object.
       def tokenize new_string=nil, options = {}
         options = @options.merge(options)
@@ -148,6 +149,11 @@ module CodeRay
       def tokens
         @cached_tokens ||= tokenize
       end
+      
+      # Whether the scanner is in streaming mode.
+      def streaming?
+        !!@options[:stream]
+      end
 
       # Traverses the tokens.
       def each &block
@@ -195,7 +201,7 @@ module CodeRay
         raise ScanError, <<-EOE % [
 
 
-***ERROR in %s: %s
+***ERROR in %s: %s (after %d tokens)
 
 tokens:
 %s
@@ -211,13 +217,14 @@ surrounding code:
 ***ERROR***
 
         EOE
-        File.basename(caller[0]),
-        msg,
-        tokens.last(10).map { |t| t.inspect }.join("\n"),
-        line, pos,
-        matched, state, bol?, eos?,
-        string[pos-ambit,ambit],
-        string[pos,ambit],
+          File.basename(caller[0]),
+          msg,
+          tokens.size,
+          tokens.last(10).map { |t| t.inspect }.join("\n"),
+          line, pos,
+          matched, state, bol?, eos?,
+          string[pos-ambit,ambit],
+          string[pos,ambit],
         ]
       end
 
-- 
cgit v1.2.1