bug#27666: [grep on GPFS filesystem] SEEK_HOLE problem

[Moyard John]

Hi,

I use GPFS file system and I have sometimes an issue using grep command.
When issue occurs with the following message "Binary file <myfile> matches"
But "<myfile>" is an ASCII one, not a binary file.
The problem seems to deals with lseek(SEEK_HOLE) command and a file not completely flushed after close.
It could take several seconds to save the entire file on the disk.

So could grep command have another way to determine if input file is binary/ASCII instead using lseek(SEEK_HOLE) ?

Best regards
john

[Eric Blake]

Post by Moyard John
Hi,
I use GPFS file system and I have sometimes an issue using grep command.
When issue occurs with the following message "Binary file <myfile> matches"
But "<myfile>" is an ASCII one, not a binary file.
The problem seems to deals with lseek(SEEK_HOLE) command and a file not completely flushed after close.

If lseek(SEEK_HOLE) returns a mid-file offset when the file is first
created, but not later after the file has been synced, then that is a
bug in the filesystem which should be reported to the appropriate
filesystem/kernel folks. SEEK_HOLE is only allowed to return a mid-file
offset if reading the file at that point in time would read NUL bytes,
and NUL bytes are indeed binary data.

Post by Moyard John
It could take several seconds to save the entire file on the disk.

Does running 'sync' prior to grep solve the problem?

[Moyard John]

Hi,

This is the kind of answer obtained from the development file system team :
---
(close(2) manpage reference)
A successful close does not guarantee that the data has been successfully saved to disk, as the kernel defers writes. It is not common for a filesystem to flush the buffers when the stream is closed. If you need to be sure that the data is physically stored, use fsync(2). (It will depend on the disk hardware at this point).
---

So running 'sync' prior to grep should solve the problem.
I don't try it yet.
Another solution found to use grep in this issue is to use activate ' --binary-files=text' grep option.

Best regards,
john



-----Message d'origine-----
De : Eric Blake [mailto:***@redhat.com]
Envoyé : mercredi 12 juillet 2017 16:11
À : Moyard John; ***@debbugs.gnu.org
Objet : Re: bug#27666: [grep on GPFS filesystem] SEEK_HOLE problem

Post by Moyard John
Hi,
I use GPFS file system and I have sometimes an issue using grep command.
When issue occurs with the following message "Binary file <myfile> matches"
But "<myfile>" is an ASCII one, not a binary file.
The problem seems to deals with lseek(SEEK_HOLE) command and a file not completely flushed after close.

If lseek(SEEK_HOLE) returns a mid-file offset when the file is first created, but not later after the file has been synced, then that is a bug in the filesystem which should be reported to the appropriate filesystem/kernel folks. SEEK_HOLE is only allowed to return a mid-file offset if reading the file at that point in time would read NUL bytes, and NUL bytes are indeed binary data.

Post by Moyard John
It could take several seconds to save the entire file on the disk.

Does running 'sync' prior to grep solve the problem?

[Moyard John]

Hi,

I forgot to precise that activate '--binary-files=text' grep option or integrate a synchronization step before a grep are not lasting solutions in all my shell scripts.
That's why I was asking about another way to identify a binary file instead using 'seek(SEEK_HOLE)' : do you think that it could possible?

Best regards
john



-----Message d'origine-----
De : Eric Blake [mailto:***@redhat.com]
Envoyé : mercredi 12 juillet 2017 16:11
À : Moyard John; ***@debbugs.gnu.org
Objet : Re: bug#27666: [grep on GPFS filesystem] SEEK_HOLE problem

Post by Moyard John
Hi,
I use GPFS file system and I have sometimes an issue using grep command.
When issue occurs with the following message "Binary file <myfile> matches"
But "<myfile>" is an ASCII one, not a binary file.
The problem seems to deals with lseek(SEEK_HOLE) command and a file not completely flushed after close.

If lseek(SEEK_HOLE) returns a mid-file offset when the file is first created, but not later after the file has been synced, then that is a bug in the filesystem which should be reported to the appropriate filesystem/kernel folks. SEEK_HOLE is only allowed to return a mid-file offset if reading the file at that point in time would read NUL bytes, and NUL bytes are indeed binary data.

Post by Moyard John
It could take several seconds to save the entire file on the disk.

Does running 'sync' prior to grep solve the problem?

[Paul Eggert]

Post by Moyard John
That's why I was asking about another way to identify a binary file instead using 'seek(SEEK_HOLE)' : do you think that it could possible?

If there is a reasonable (i.e., cheap) way for grep to determine that
SEEK_HOLE is buggy for the current file, I suppose grep could do that.
Do you know of any such method?

Really, the bug here is in the file system, not in grep. Have you filed
a bug with the GPFS maintainers?

[Moyard John]

GPFS maintainers give me the answer including manpage close(2) : nothing will be done.
On the web, same problems has been identified for ZFS or perhaps NFS v4 ? :
https://utcc.utoronto.ca/~cks/space/blog/linux/GrepBinaryFileReason
https://github.com/zfsonlinux/zfs/issues/6050
https://lists.gnu.org/archive/html/bug-grep/2012-07/msg00022.html
I will not file a bug on each file system maintainers : I should obtain the same answer.
Or perhaps I will obtain an extract of manpage lseek(2), i.e. http://man7.org/linux/man-pages/man2/lseek.2.html :
However, a filesystem is not obliged to report holes, so
these operations are not a guaranteed mechanism for mapping the
storage space actually allocated to a file
It's not a bug in file system.

So, is-it possible for you to modify something about the way to test binary file ?

john

-----Message d'origine-----
De : Paul Eggert [mailto:***@cs.ucla.edu]
Envoyé : jeudi 13 juillet 2017 22:44
À : Moyard John; Eric Blake; ***@debbugs.gnu.org
Objet : Re: bug#27666: [grep on GPFS filesystem] SEEK_HOLE problem

Post by Moyard John
That's why I was asking about another way to identify a binary file instead using 'seek(SEEK_HOLE)' : do you think that it could possible?

If there is a reasonable (i.e., cheap) way for grep to determine that SEEK_HOLE is buggy for the current file, I suppose grep could do that.
Do you know of any such method?

Really, the bug here is in the file system, not in grep. Have you fi

[Eric Blake]

Post by Moyard John
GPFS maintainers give me the answer including manpage close(2) : nothing will be done.
https://utcc.utoronto.ca/~cks/space/blog/linux/GrepBinaryFileReason
https://github.com/zfsonlinux/zfs/issues/6050
https://lists.gnu.org/archive/html/bug-grep/2012-07/msg00022.html
I will not file a bug on each file system maintainers : I should obtain the same answer.
However, a filesystem is not obliged to report holes, so
these operations are not a guaranteed mechanism for mapping the
storage space actually allocated to a file
It's not a bug in file system.

A file system is not obliged to report holes, but IS obliged to NOT
report holes if a read() on that range will not see zeroes. I still
think GPFS has a bug.

[Paul Eggert]

Post by Moyard John
GPFS maintainers give me the answer including manpage close(2) : nothing will be done.

Sorry, I don't follow (why is close(2) involved?).

Is your correspondence with the GPFS maintainers public? It sounds like they do
not understand the issue.

Anyway, as Eric said, GPFS is clearly buggy. True, a file system is not obliged
to report holes. But if it reports a hole, the hole must contain NUL bytes.

Post by Moyard John
https://utcc.utoronto.ca/~cks/space/blog/linux/GrepBinaryFileReason
https://github.com/zfsonlinux/zfs/issues/6050

These URLs talk about a ZFS-on-Linux bug that has been fixed, apparently. Good.

Post by Moyard John
https://lists.gnu.org/archive/html/bug-grep/2012-07/msg00022.html

This is the inverse issue, which doesn't cause the problem you mentioned.

Post by Moyard John
I will not file a bug on each file system maintainers : I should obtain the same answer.

I don't see why. Only GPFS has the problem, as far as we know. And this is
probably just a communication problem with its developers.

Post by Moyard John
So, is-it possible for you to modify something about the way to test binary file ?

Programs other than 'grep' use SEEK_HOLE. Even if we changed 'grep' to stop
using SEEK_HOLE, the other programs would still be broken on GPFS. Plus, 'grep'
would likely be slower everywhere, just to work around the bug on GPFS.

Really, GPFS needs to be fixed. If GPFS can't support SEEK_HOLE correctly, it
should simply have lseek with SEEK_HOLE go to end-of-file; that will work with
'grep' (albeit more slowly), and is the documented way that SEEK_HOLE is
supposed to work on file systems that cannot support SEEK_HOLE directly.

[Moyard John]

Thank your very much for your detailed answer.
"close(2)" is involved because a test case made to reproduce the problem use a cp, initially a fortran code to make a copy, follow by a grep.

I clearly understand your point of view about
reporting hole and NUL bytes
GPFS incompatibility with others programs/commands that could use SEEK_HOLE
I try to take a quick look about this last point and don't find yet any system command using it.
Do you have an example of other command using SEEK_HOLE?

In POSIX point of view, lseek(2) manpage precise this :
SEEK_DATA and SEEK_HOLE are nonstandard extensions also present in Solaris, FreeBSD, and DragonFly BSD
They are proposed for inclusion in the next POSIX revision (Issue 8)
Do you have any information about it?
Does compile 'grep' mechanism could avoid the use of SEEK_HOLE test ?
I just try to obtain a grep command with a default behavior in respect of POSIX standard.



-----Message d'origine-----
De : Paul Eggert [mailto:***@cs.ucla.edu]
Envoyé : mercredi 19 juillet 2017 12:48
À : Moyard John; Eric Blake; ***@debbugs.gnu.org
Objet : Re: bug#27666: [grep on GPFS filesystem] SEEK_HOLE problem

Post by Moyard John
GPFS maintainers give me the answer including manpage close(2) : nothing will be done.

Sorry, I don't follow (why is close(2) involved?).

Is your correspondence with the GPFS maintainers public? It sounds like they do not understand the issue.

Anyway, as Eric said, GPFS is clearly buggy. True, a file system is not obliged to report holes. But if it reports a hole, the hole must contain NUL bytes.

Post by Moyard John
https://utcc.utoronto.ca/~cks/space/blog/linux/GrepBinaryFileReason
https://github.com/zfsonlinux/zfs/issues/6050

These URLs talk about a ZFS-on-Linux bug that has been fixed, apparently. Good.

Post by Moyard John
https://lists.gnu.org/archive/html/bug-grep/2012-07/msg00022.html

This is the inverse issue, which doesn't cause the problem you mentioned.

Post by Moyard John
I will not file a bug on each file system maintainers : I should obtain the same answer.

I don't see why. Only GPFS has the problem, as far as we know. And this is probably just a communication problem with its developers.

Post by Moyard John
So, is-it possible for you to modify something about the way to test binary file ?

Programs other than 'grep' use SEEK_HOLE. Even if we changed 'grep' to stop using SEEK_HOLE, the other programs would still be broken on GPFS. Plus, 'grep'
would likely be slower everywhere, just to work around the bug on GPFS.

Really, GPFS needs to be fixed. If GPFS can't support SEEK_HOLE correctly, it should simply have lseek with SEEK_HOLE go to end-of-file; that will work with 'grep' (albeit more slowly), and is the documented way that SEEK_HOLE is supposed to work on file systems that cannot support

[Eric Blake]

Post by Moyard John
Thank your very much for your detailed answer.
"close(2)" is involved because a test case made to reproduce the problem use a cp, initially a fortran code to make a copy, follow by a grep.
I clearly understand your point of view about
reporting hole and NUL bytes
GPFS incompatibility with others programs/commands that could use SEEK_HOLE
I try to take a quick look about this last point and don't find yet any system command using it.
Do you have an example of other command using SEEK_HOLE?

More and more commands are starting to make optimizations based on
SEEK_HOLE. cp, tar, diff, grep, etc. Programs like qemu-img REQUIRE a
working SEEK_HOLE for efficiently managing sparse virtual machine disk
images.

Post by Moyard John
SEEK_DATA and SEEK_HOLE are nonstandard extensions also present in Solaris, FreeBSD, and DragonFly BSD
They are proposed for inclusion in the next POSIX revision (Issue 8)
Do you have any information about it?

Here's the proposed POSIX wording:
http://austingroupbugs.net/view.php?id=415

Requiring close() to occur before SEEK_HOLE is accurate is a bug in GPFS
(if any other process can read() non-zero data but lseek(SEEK_HOLE)
still claims that section of the file is a hole, then the file system is
buggy, per the wording POSIX will be adding).

Post by Moyard John
Does compile 'grep' mechanism could avoid the use of SEEK_HOLE test ?

No. Avoiding a buggy SEEK_HOLE in grep won't fix all the other programs
(like cp, tar, diff) that are also negatively impacted by the buggy
SEEK_HOLE. Fix the GPFS bug, and then all of the user-space apps will
no longer be impacted by the bug.

[By the way, top-posting is frowned on for technical lists]. I agree

Post by Moyard John
Really, GPFS needs to be fixed. If GPFS can't support SEEK_HOLE correctly, it should simply have lseek with SEEK_HOLE go to end-of-file; that will work with 'grep' (albeit more slowly), and is the documented way that SEEK_HOLE is supposed to work on file systems that cannot support SEEK_HOLE directly.