Read Perl code on Windows in text mode by default.

We used to read Perl code in binary mode to make life easier for
ByteLoder to include binary data in a source file.  To maintain the
illusion of text mode for the DATA handle the filehandle was
transformed from binary mode to text mode when the parser reached the
__END__ or __DATA__ tokens.

This however never worked correctly, as the positions returned by
tell(DATA) were still based on reading part of the stream in binary
mode.  And even worse, flushing all filehandles before calling
system(), backticks, or fork() would actually reposition the DATA
filehandle incorrectly, so future reads from it returned the wrong
data.

http://rt.perl.org/rt3/Ticket/Display.html?id=28106 contains several
bug reports that are all related to this problem.  The new t/io/data.t
file contains the failing code samples from those bugs.

This patch changes the default build option for Windows to text mode.
ByteLoader will have to deal with this internally, e.g. by rewinding
DATA and switching to binary mode itself.

5 files changed, 112 insertions, 6 deletions

diff --git a/MANIFEST b/MANIFEST
index 2e62c32647..f8d1184fbf 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -4682,6 +4682,7 @@ t/io/argv.t			See if ARGV stuff works
 t/io/binmode.t			See if binmode() works
 t/io/crlf.t			See if :crlf works
 t/io/crlf_through.t		See if pipe passes data intact with :crlf
+t/io/data.t			See if DATA works
 t/io/defout.t			See if PL_defoutgv works
 t/io/dup.t			See if >& works right
 t/io/eintr.t			See if code called during EINTR is safe
diff --git a/pod/perldelta.pod b/pod/perldelta.pod
index b6ae3f7123..d71512c5b2 100644
--- a/pod/perldelta.pod
+++ b/pod/perldelta.pod
@@ -91,6 +91,18 @@ can't be blocked on I/O either.
 See L<perlfork> for more information about the fork() emulation on
 Windows.
 
+=head2 Perl source code is read in text mode on Windows
+
+Perl scripts used to be read in binary mode on Windows for the benefit
+of the ByteLoader module (which is no longer part of core Perl).  This
+had the side effect of breaking various operations on the DATA filehandle,
+including seek()/tell(), and even simply reading from DATA after file handles
+have been flushed by a call to system(), backticks, fork() etc.
+
+The default build options for Windows have been changed to read Perl source
+code on Windows in text mode now.  Hopefully ByteLoader will be updated on
+CPAN to automatically handle this situation.
+
 =head1 Deprecations
 
 XXX Any deprecated features, syntax, modules etc. should be listed here.
diff --git a/t/io/data.t b/t/io/data.t
new file mode 100644
index 0000000000..52c6bc03c7
--- /dev/null
+++ b/t/io/data.t
@@ -0,0 +1,81 @@
+#!./perl
+
+# tests for DATA filehandle operations
+
+BEGIN {
+    chdir 't' if -d 't';
+    @INC = '../lib';
+    require './test.pl';
+}
+
+$|=1;
+
+# It is important that all these tests are run via fresh_perl because
+# that way they get written to disk in text mode and will have CR-LF
+# line endings on Windows.  Otherwise the failures related to Perl
+# code being read in binary mode will not be observed.
+
+run_multiple_progs('', \*DATA);
+
+done_testing();
+
+__END__
+# http://rt.perl.org/rt3/Ticket/Display.html?id=28106#txn-82657
+while (<DATA>) {
+    chomp;
+    print "$.: '$_'\n";
+    system();
+}
+__DATA__
+1
+2
+3
+EXPECT
+1: '1'
+2: '2'
+3: '3'
+########
+# http://rt.perl.org/rt3/Ticket/Display.html?id=28106#txn-83113
+my $line1 = <DATA>;
+`echo foo`;
+my $line2 = <DATA>;
+if ($line1 eq "one\n") { print "ok 1\n" } else { print "not ok 1\n" }
+if ($line2 eq "two\n") { print "ok 2\n" } else { print "not ok 2\n" }
+__DATA__
+one
+two
+EXPECT
+ok 1
+ok 2
+########
+# http://rt.perl.org/rt3/Ticket/Attachment/828796/403048/perlbug.rep.txt
+my @data_positions = tell(DATA);
+while (<DATA>){
+    if (/^__DATA__$/) {
+        push @data_positions, tell(DATA);
+    }
+}
+
+my @fh_positions;
+open(my $fh, '<', $0) or die;
+while (<$fh>){
+    if (/^__DATA__$/) {
+        push @fh_positions, tell($fh);
+    }
+}
+
+print "not " unless "@data_positions" eq "@fh_positions";
+print "ok";
+
+__DATA__
+ab
+__DATA__
+ab
+
+__DATA__
+ab
+__DATA__
+lotsa junk
+nothing
+EXPECT
+ok
diff --git a/win32/Makefile b/win32/Makefile
index e9ca59fe7d..6fecf5ff54 100644
--- a/win32/Makefile
+++ b/win32/Makefile
@@ -190,10 +190,16 @@ BUILDOPT	= $(BUILDOPTEXTRA)
 #BUILDOPT	= $(BUILDOPT) -DPERL_EXTERNAL_GLOB
 
 #
-# This should normally be disabled.  Enabling it causes perl to read scripts
-# in text mode (which is the 5.005 behavior) and will break ByteLoader.
+# Perl needs to read scripts in text mode so that the DATA filehandle
+# works correctly with seek() and tell(), or around auto-flushes of
+# all filehandles (e.g. by system(), backticks, fork(), etc).
 #
-#BUILDOPT	= $(BUILDOPT) -DPERL_TEXTMODE_SCRIPTS
+# The current version on the ByteLoader module on CPAN however only
+# works if scripts are read in binary mode.  But before you disable text
+# mode script reading (and break some DATA filehandle functionality)
+# please check first if an updated ByteLoader isn't available on CPAN.
+#
+BUILDOPT	= $(BUILDOPT) -DPERL_TEXTMODE_SCRIPTS
 
 #
 # specify semicolon-separated list of extra directories that modules will
diff --git a/win32/makefile.mk b/win32/makefile.mk
index b62c27c7b8..f30906032e 100644
--- a/win32/makefile.mk
+++ b/win32/makefile.mk
@@ -246,10 +246,16 @@ BUILDOPT	*= $(BUILDOPTEXTRA)
 #BUILDOPT	+= -DPERL_EXTERNAL_GLOB
 
 #
-# This should normally be disabled.  Enabling it causes perl to read scripts
-# in text mode (which is the 5.005 behavior) and will break ByteLoader.
+# Perl needs to read scripts in text mode so that the DATA filehandle
+# works correctly with seek() and tell(), or around auto-flushes of
+# all filehandles (e.g. by system(), backticks, fork(), etc).
 #
-#BUILDOPT	+= -DPERL_TEXTMODE_SCRIPTS
+# The current version on the ByteLoader module on CPAN however only
+# works if scripts are read in binary mode.  But before you disable text
+# mode script reading (and break some DATA filehandle functionality)
+# please check first if an updated ByteLoader isn't available on CPAN.
+#
+BUILDOPT	+= -DPERL_TEXTMODE_SCRIPTS
 
 #
 # specify semicolon-separated list of extra directories that modules will