delta/gnulib.git/modules/fts, branch python git.savannah.gnu.org: git/gnulib.git fts: fix cloexec races 2017-08-12T18:36:10+00:00 Paul Eggert eggert@cs.ucla.edu 2017-08-12T18:20:41+00:00 b1f55699e4ca1e66987385edeff6859672e5a070 * lib/fts.c [!_LIBC]: Do not include dirent--.h, unistd--.h, cloexec.h. (opendirat, diropen): Use O_CLOEXEC instead of set_cloexec_flag. (fts_build): Use F_DUPD_CLOEXEC rinstad of set_cloexec_flag. (fd_ring_check): Set cloexec flag on new file descriptors. (fts_build, fd_ring_check): While we’re at it, make sure the resulting file descriptor is not 0, 1, or 2, since that is easy. * modules/fts (Depends-on): Remove cloexec, dirent-safer, dup, fcntl-safer, unistd-safer. Add fcntl. 
* lib/fts.c [!_LIBC]: Do not include dirent--.h, unistd--.h, cloexec.h.
(opendirat, diropen): Use O_CLOEXEC instead of set_cloexec_flag.
(fts_build): Use F_DUPD_CLOEXEC rinstad of set_cloexec_flag.
(fd_ring_check): Set cloexec flag on new file descriptors.
(fts_build, fd_ring_check): While we’re at it, make sure the
resulting file descriptor is not 0, 1, or 2, since that is easy.
* modules/fts (Depends-on): Remove cloexec, dirent-safer, dup,
fcntl-safer, unistd-safer.  Add fcntl.
fts: port to C11 alignof 2015-10-18T17:04:12+00:00 Paul Eggert eggert@cs.ucla.edu 2015-10-18T16:52:45+00:00 04dee58e17db70e8d0396166be00e6a1b38c31d7 * doc/posix-headers/stdalign.texi (stdalign.h): Document the C11 restriction. * lib/fts.c: Include stddef.h, for max_align_t. (fts_alloc): Align using max_align_t, not FTSENT. * modules/fts (Depends-on): Add stddef. 
* doc/posix-headers/stdalign.texi (stdalign.h):
Document the C11 restriction.
* lib/fts.c: Include stddef.h, for max_align_t.
(fts_alloc): Align using max_align_t, not FTSENT.
* modules/fts (Depends-on): Add stddef.
fts: avoid reading beyond the heap allocation 2015-06-25T01:35:59+00:00 Pádraig Brady P@draigBrady.com 2015-06-24T22:52:24+00:00 49078a780041205fbbab56802033595eb44f854d GCC 5.1.1 with -O2 and -fsanitize=address reports the following from `chmod a+rx ..` for example: ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61200000be48 READ of size 4 at 0x61200000be48 thread T0 #0 0x40f0a1 in fts_stat lib/fts.c:1821 #1 0x4141ec in fts_open lib/fts.c:514 #2 0x40ca6e in xfts_open lib/xfts.c:36 #3 0x402c35 in process_files src/chmod.c:336 #4 0x402c35 in main src/chmod.c:566 #5 0x7f8fdc38678f in __libc_start_main (/lib64/libc.so.6+0x2078f) #6 0x404608 in _start (/home/padraig/git/coreutils/src/chmod+0x404608) 0x61200000be4b is located 0 bytes to the right of 267-byte region [0x61200000bd40,0x61200000be4b) allocated by thread T0 here: #0 0x7f8fdd4cfa0a in malloc (/lib64/libasan.so.2+0x98a0a) #1 0x40f22c in fts_alloc lib/fts.c:1916 The read of size 4 from a heap object of size 3 is indeed invalid, though this may be due to incorrect padding assumptions by GCC, see: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66661 The read is coming from the a[2] access in ISDOT() which GCC seems to assume can access according to the alignment of the main structure, since the flexible array members are being accessed directly. One could cast the parameter to ISDOT() to (char const*) to restrict to byte at a time access, however it seems more correct and maintainable to increase the buffer size to the appropriate alignment to avoid this issue. For reference some notes on alignment and flexible array members are at: https://sites.google.com/site/embeddedmonologue/home/c-programming/data-alignment-structure-padding-and-flexible-array-member * lib/fts.c (fts_alloc): Increase allocation to alignof(FTSENT). * modules/fts: Depend on stdalign. 
GCC 5.1.1 with -O2 and -fsanitize=address reports
the following from `chmod a+rx ..` for example:

  ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61200000be48
  READ of size 4 at 0x61200000be48 thread T0
    #0 0x40f0a1 in fts_stat lib/fts.c:1821
    #1 0x4141ec in fts_open lib/fts.c:514
    #2 0x40ca6e in xfts_open lib/xfts.c:36
    #3 0x402c35 in process_files src/chmod.c:336
    #4 0x402c35 in main src/chmod.c:566
    #5 0x7f8fdc38678f in __libc_start_main (/lib64/libc.so.6+0x2078f)
    #6 0x404608 in _start (/home/padraig/git/coreutils/src/chmod+0x404608)

  0x61200000be4b is located 0 bytes to the right of
  267-byte region [0x61200000bd40,0x61200000be4b) allocated by thread T0 here:
    #0 0x7f8fdd4cfa0a in malloc (/lib64/libasan.so.2+0x98a0a)
    #1 0x40f22c in fts_alloc lib/fts.c:1916

The read of size 4 from a heap object of size 3 is indeed invalid,
though this may be due to incorrect padding assumptions by GCC, see:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66661

The read is coming from the a[2] access in ISDOT() which GCC seems
to assume can access according to the alignment of the main structure,
since the flexible array members are being accessed directly.
One could cast the parameter to ISDOT() to (char const*) to restrict
to byte at a time access, however it seems more correct and maintainable
to increase the buffer size to the appropriate alignment to avoid this issue.
For reference some notes on alignment and flexible array members are at:
https://sites.google.com/site/embeddedmonologue/home/c-programming/data-alignment-structure-padding-and-flexible-array-member

* lib/fts.c (fts_alloc): Increase allocation to alignof(FTSENT).
* modules/fts: Depend on stdalign.
fts: port to GCC 5.1 with --enable-gcc-warnings 2015-04-27T08:24:40+00:00 Paul Eggert eggert@cs.ucla.edu 2015-04-27T08:19:03+00:00 e4584a9e26ea39e2f961a700c60f3a10ec07dbb4 Without this fix, GCC 5.1 (correctly) warns about a subscript error on the fts_name component of FTSENT. It's actually a flexible member, so define it that way on C99 or later hosts. * lib/fts.c (fts_alloc): Use offsetof, not sizeof, for a structure that now has a flexible array member. * lib/fts_.h (__FLEXIBLE_ARRAY_MEMBER): New macro. (FTSENT): fts_name is now flexible on C99-or-later platforms. * modules/fts (Depends-on): Add flexmember. 
Without this fix, GCC 5.1 (correctly) warns about a subscript
error on the fts_name component of FTSENT.  It's actually a
flexible member, so define it that way on C99 or later hosts.
* lib/fts.c (fts_alloc): Use offsetof, not sizeof, for a
structure that now has a flexible array member.
* lib/fts_.h (__FLEXIBLE_ARRAY_MEMBER): New macro.
(FTSENT): fts_name is now flexible on C99-or-later platforms.
* modules/fts (Depends-on): Add flexmember.
fts: depend on fdopendir 2012-03-10T20:10:44+00:00 Paul Eggert eggert@cs.ucla.edu 2012-03-10T20:10:17+00:00 ef693061d753a982cf1bdd627b1d1c158f54ba06 * modules/fts (Depends-on): Depend on fdopendir. This is needed on Solaris 8, at least, since it lacks fdopendir. Evidently the problem was introduced when fdopendir was split out. 
* modules/fts (Depends-on): Depend on fdopendir.  This is needed
on Solaris 8, at least, since it lacks fdopendir.  Evidently the
problem was introduced when fdopendir was split out.
New modules 'at-internal', 'openat-h', split off from module 'openat'. 2011-11-05T10:35:57+00:00 Bruno Haible bruno@clisp.org 2011-11-05T10:35:57+00:00 75000c499c834249ea7be7a0aa2799a817d5b58c * m4/openat.m4 (gl_FUNC_OPENAT): Don't set GNULIB_OPENAT. Don't invoke gl_PREREQ_OPENAT and gl_MODULE_INDICATOR. * modules/openat (Description): Add reference to POSIX function. (Files): Remove lib/openat.h, lib/openat-proc.c. (Depends-on): Add at-internal, openat-h. Remove fdopendir, gettext-h, intprops, unistd. (configure.ac): Remove AC_LIBOBJ of openat-proc. Invoke gl_PREREQ_OPENAT, gl_MODULE_INDICATOR here. Invoke gl_FCNTL_MODULE_INDICATOR. (Include): Remove unistd.h, openat.h. * modules/areadlinkat (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, extensions, fchdir, openat-die, openat-h, save-cwd. * modules/areadlinkat-with-size (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, extensions, fchdir, openat-die, openat-h, save-cwd, unistd. * modules/faccessat (Files): Add lib/at-func.c, lib/openat-priv.h. (Depends-on): Add at-internal, dosname, errno, fchdir, openat-die, openat-h, save-cwd. Remove fcntl-h, openat. * modules/fchmodat (Files): Remove lib/openat.h. (Depends-on): Add at-internal, openat-h. Remove dirent, fcntl-h, openat, stdbool, unistd. * modules/fchownat (Files): Remove lib/openat.h. (Depends-on): Add at-internal, openat-h. Remove dirent, fcntl-h, openat, stdbool, sys_stat. * modules/fdopendir (Files): Remove lib/openat-priv.h, lib/openat-proc.c. (Depends-on): Add at-internal. (condigure.ac): Remove AC_LIBOBJ of openat-proc. * modules/fstatat (Files): Remove lib/openat.h. (Depends-on): Add at-internal, openat-h. Remove dirent, openat, stdbool, unistd. * modules/fts (Depends-on): Add openat-h. * modules/linkat (Depends-on): Add at-internal, openat-h. Remove openat. * modules/mkdirat (Files): Remove lib/openat.h. (Depends-on): Add at-internal, openat-h. Remove dirent, fcntl-h, openat, stdbool, sys_stat. * modules/mkfifoat (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, fchdir, openat-die, openat-h, save-cwd. Remove fcntl-h, openat. * modules/openat-die (Depends-on): Add openat-h. Remove openat. * modules/readlinkat (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, fchdir, openat-die, openat-h, save-cwd. Remove fcntl-h, openat. * modules/renameat (Depends-on): Add at-internal, openat-h. Remove openat. * modules/selinux-at (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, extensions, fchdir, fcntl-h, openat-die, openat-h, save-cwd, unistd. Remove openat. * modules/symlinkat (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, fchdir, openat-die, openat-h, save-cwd. Remove fcntl-h, openat. * modules/unlinkat (Files): Remove lib/openat.h. (Depends-on): Add at-internal, openat-h. Remove dirent, openat, stdbool. * modules/utimensat (Files): Add lib/at-func.c. (Depends-on): Add at-internal, dosname, errno, fchdir, fcntl-h, openat-die, openat-h, save-cwd. * modules/fchownat-tests (Depends-on): Add openat-h. Remove openat. * modules/fdutimensat-tests (Depends-on): Add openat. * modules/fstatat-tests (Depends-on): Add openat-h. * modules/readlinkat-tests (Depends-on): Add openat. * modules/symlinkat-tests (Depends-on): Add openat. 
* m4/openat.m4 (gl_FUNC_OPENAT): Don't set GNULIB_OPENAT. Don't
invoke gl_PREREQ_OPENAT and gl_MODULE_INDICATOR.
* modules/openat (Description): Add reference to POSIX function.
(Files): Remove lib/openat.h, lib/openat-proc.c.
(Depends-on): Add at-internal, openat-h. Remove fdopendir, gettext-h,
intprops, unistd.
(configure.ac): Remove AC_LIBOBJ of openat-proc. Invoke
gl_PREREQ_OPENAT, gl_MODULE_INDICATOR here. Invoke
gl_FCNTL_MODULE_INDICATOR.
(Include): Remove unistd.h, openat.h.
* modules/areadlinkat (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, extensions, fchdir,
openat-die, openat-h, save-cwd.
* modules/areadlinkat-with-size (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, extensions, fchdir,
openat-die, openat-h, save-cwd, unistd.
* modules/faccessat (Files): Add lib/at-func.c, lib/openat-priv.h.
(Depends-on): Add at-internal, dosname, errno, fchdir, openat-die,
openat-h, save-cwd. Remove fcntl-h, openat.
* modules/fchmodat (Files): Remove lib/openat.h.
(Depends-on): Add at-internal, openat-h. Remove dirent, fcntl-h,
openat, stdbool, unistd.
* modules/fchownat (Files): Remove lib/openat.h.
(Depends-on): Add at-internal, openat-h. Remove dirent, fcntl-h,
openat, stdbool, sys_stat.
* modules/fdopendir (Files): Remove lib/openat-priv.h,
lib/openat-proc.c.
(Depends-on): Add at-internal.
(condigure.ac): Remove AC_LIBOBJ of openat-proc.
* modules/fstatat (Files): Remove lib/openat.h.
(Depends-on): Add at-internal, openat-h. Remove dirent, openat,
stdbool, unistd.
* modules/fts (Depends-on): Add openat-h.
* modules/linkat (Depends-on): Add at-internal, openat-h. Remove
openat.
* modules/mkdirat (Files): Remove lib/openat.h.
(Depends-on): Add at-internal, openat-h. Remove dirent, fcntl-h,
openat, stdbool, sys_stat.
* modules/mkfifoat (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, fchdir, openat-die,
openat-h, save-cwd. Remove fcntl-h, openat.
* modules/openat-die (Depends-on): Add openat-h. Remove openat.
* modules/readlinkat (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, fchdir, openat-die,
openat-h, save-cwd. Remove fcntl-h, openat.
* modules/renameat (Depends-on): Add at-internal, openat-h. Remove
openat.
* modules/selinux-at (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, extensions, fchdir,
fcntl-h, openat-die, openat-h, save-cwd, unistd. Remove openat.
* modules/symlinkat (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, fchdir, openat-die,
openat-h, save-cwd. Remove fcntl-h, openat.
* modules/unlinkat (Files): Remove lib/openat.h.
(Depends-on): Add at-internal, openat-h. Remove dirent, openat,
stdbool.
* modules/utimensat (Files): Add lib/at-func.c.
(Depends-on): Add at-internal, dosname, errno, fchdir, fcntl-h,
openat-die, openat-h, save-cwd.
* modules/fchownat-tests (Depends-on): Add openat-h. Remove openat.
* modules/fdutimensat-tests (Depends-on): Add openat.
* modules/fstatat-tests (Depends-on): Add openat-h.
* modules/readlinkat-tests (Depends-on): Add openat.
* modules/symlinkat-tests (Depends-on): Add openat.
New module 'fstat'. 2011-09-25T11:42:08+00:00 Bruno Haible bruno@clisp.org 2011-09-25T11:42:08+00:00 cd76a061a80d91823660fc999727c6ec5f4a067b * lib/sys_stat.in.h (fstat): Declare only if GNULIB_FSTAT is set. * lib/fstat.c: New file, based on a piece of lib/fchdir.c. * lib/fchdir.c (rpl_fstat): Remove function. * m4/fstat.m4: New file. * m4/fchdir.m4 (gl_FUNC_FCHDIR): Don't set REPLACE_FSTAT. * m4/sys_stat_h.m4 (gl_HEADER_SYS_STAT_H): Test whether fstat is declared. (gl_SYS_STAT_H_DEFAULTS): Initialize GNULIB_FSTAT. * modules/sys_stat (Makefile.am): Substitute GNULIB_FSTAT. * modules/fstat: New file. * modules/sys_stat-tests (Depends-on): Remove fstat-tests. * tests/test-sys_stat-c++.cc (fstat): Check only if GNULIB_TEST_FSTAT is set. * doc/posix-functions/fstat.texi: Mention the new module and the problem on MSVC. * NEWS: Mention the change. * modules/acl (Depends-on): Add fstat. * modules/chdir-safer (Depends-on): Likewise. * modules/chown (Depends-on): Likewise. * modules/copy-file (Depends-on): Likewise. * modules/fchdir (Depends-on): Likewise. * modules/fdopendir (Depends-on): Likewise. * modules/fopen (Depends-on): Likewise. * modules/fts (Depends-on): Likewise. * modules/getcwd (Depends-on): Likewise. * modules/isapipe (Depends-on): Likewise. * modules/linkat (Depends-on): Likewise. * modules/lseek (Depends-on): Likewise. * modules/mkdir-p (Depends-on): Likewise. * modules/open (Depends-on): Likewise. * modules/openat (Depends-on): Likewise. * modules/read-file (Depends-on): Likewise. * modules/renameat (Depends-on): Likewise. * modules/utimens (Depends-on): Likewise. 
* lib/sys_stat.in.h (fstat): Declare only if GNULIB_FSTAT is set.
* lib/fstat.c: New file, based on a piece of lib/fchdir.c.
* lib/fchdir.c (rpl_fstat): Remove function.
* m4/fstat.m4: New file.
* m4/fchdir.m4 (gl_FUNC_FCHDIR): Don't set REPLACE_FSTAT.
* m4/sys_stat_h.m4 (gl_HEADER_SYS_STAT_H): Test whether fstat is
declared.
(gl_SYS_STAT_H_DEFAULTS): Initialize GNULIB_FSTAT.
* modules/sys_stat (Makefile.am): Substitute GNULIB_FSTAT.
* modules/fstat: New file.
* modules/sys_stat-tests (Depends-on): Remove fstat-tests.
* tests/test-sys_stat-c++.cc (fstat): Check only if GNULIB_TEST_FSTAT
is set.
* doc/posix-functions/fstat.texi: Mention the new module and the
problem on MSVC.
* NEWS: Mention the change.
* modules/acl (Depends-on): Add fstat.
* modules/chdir-safer (Depends-on): Likewise.
* modules/chown (Depends-on): Likewise.
* modules/copy-file (Depends-on): Likewise.
* modules/fchdir (Depends-on): Likewise.
* modules/fdopendir (Depends-on): Likewise.
* modules/fopen (Depends-on): Likewise.
* modules/fts (Depends-on): Likewise.
* modules/getcwd (Depends-on): Likewise.
* modules/isapipe (Depends-on): Likewise.
* modules/linkat (Depends-on): Likewise.
* modules/lseek (Depends-on): Likewise.
* modules/mkdir-p (Depends-on): Likewise.
* modules/open (Depends-on): Likewise.
* modules/openat (Depends-on): Likewise.
* modules/read-file (Depends-on): Likewise.
* modules/renameat (Depends-on): Likewise.
* modules/utimens (Depends-on): Likewise.
New module 'dup'. 2011-09-24T00:08:13+00:00 Bruno Haible bruno@clisp.org 2011-09-24T00:08:13+00:00 4810e2dc1d9b1bf84da97d7869f7ad8743b3b3c4 * lib/unistd.in.h (dup): Declare only if the 'dup' module is in use. Allow replacement. * lib/dup.c: New file. * lib/fchdir.c (rpl_dup): Remove function. * m4/dup.m4: New file. * m4/fchdir.m4 (gl_FUNC_FCHDIR): Don't set REPLACE_DUP here. * m4/unistd_h.m4 (gl_UNISTD_H): Test whether 'dup' is declared. (gl_UNISTD_H_DEFAULTS): Initialize GNULIB_DUP. * modules/unistd (Makefile.am): Substitute GNULIB_DUP. * modules/dup: New file. * tests/test-unistd-c++.cc: Check the signature of 'dup' only if the 'dup' module is in use. * modules/fdopendir (Depends-on): Add dup. * modules/fdutimensat-tests (Depends-on): Likewise. * modules/fts (Depends-on): Likewise. * modules/futimens-tests (Depends-on): Likewise. * modules/posix_spawnp-tests (Depends-on): Likewise. * modules/unistd-safer-tests (Depends-on): Likewise. * modules/utimens-tests (Depends-on): Likewise. * doc/posix-functions/dup.texi: Mention the new module and the problem on MSVC. 
* lib/unistd.in.h (dup): Declare only if the 'dup' module is in use.
Allow replacement.
* lib/dup.c: New file.
* lib/fchdir.c (rpl_dup): Remove function.
* m4/dup.m4: New file.
* m4/fchdir.m4 (gl_FUNC_FCHDIR): Don't set REPLACE_DUP here.
* m4/unistd_h.m4 (gl_UNISTD_H): Test whether 'dup' is declared.
(gl_UNISTD_H_DEFAULTS): Initialize GNULIB_DUP.
* modules/unistd (Makefile.am): Substitute GNULIB_DUP.
* modules/dup: New file.
* tests/test-unistd-c++.cc: Check the signature of 'dup' only if the
'dup' module is in use.
* modules/fdopendir (Depends-on): Add dup.
* modules/fdutimensat-tests (Depends-on): Likewise.
* modules/fts (Depends-on): Likewise.
* modules/futimens-tests (Depends-on): Likewise.
* modules/posix_spawnp-tests (Depends-on): Likewise.
* modules/unistd-safer-tests (Depends-on): Likewise.
* modules/utimens-tests (Depends-on): Likewise.
* doc/posix-functions/dup.texi: Mention the new module and the problem
on MSVC.
Add dependencies to new dirent related modules. 2011-09-17T14:53:15+00:00 Bruno Haible bruno@clisp.org 2011-09-17T14:53:15+00:00 7943c52181dd2ff0e9c02b14c5a504da09db337b * modules/backupfile (Depends-on): Add opendir, readdir, closedir. * modules/fts (Depends-on): Likewise. * modules/glob (Depends-on): Likewise. * modules/savedir (Depends-on): Likewise. * modules/scandir (Depends-on): Likewise. * modules/dirent-safer (Depends-on): Add opendir, closedir. * modules/fdopendir (Depends-on): Add opendir. 
* modules/backupfile (Depends-on): Add opendir, readdir, closedir.
* modules/fts (Depends-on): Likewise.
* modules/glob (Depends-on): Likewise.
* modules/savedir (Depends-on): Likewise.
* modules/scandir (Depends-on): Likewise.
* modules/dirent-safer (Depends-on): Add opendir, closedir.
* modules/fdopendir (Depends-on): Add opendir.
fts: Move AC_LIBOBJ invocations to module description. 2011-06-15T22:07:25+00:00 Bruno Haible bruno@clisp.org 2011-05-23T21:31:06+00:00 4b40eab91d0e7b3aaf69ba51ab236c079c904649 * m4/fts.m4 (gl_FUNC_FTS_CORE): Move AC_LIBOBJ invocation from here... * modules/fts (configure.ac): ... to here. 
* m4/fts.m4 (gl_FUNC_FTS_CORE): Move AC_LIBOBJ invocation from here...
* modules/fts (configure.ac): ... to here.