Some serious issues with the new -O option
 | Classic |  | List |  | Threaded |
123
123
64 posts
|
I'm sorry to say that the new -O option can interact *badly*
with Automake-generated parallel testsuites (at least when they are being run "interactively", that is, with the output of make connected to a tty). Let me elaborate a little (if you still have doubts or objections after reading the considerations below, please do not hesitate to ask for clarifications). For long-running testsuite with lots of tests, the use of the '-O' option prevents the results from the tests already run to be displayed on the screen until the *whole* testsuite has run. IMO, this destroys feedback and readability of the output. Try building GNU coreutils and running its testsuite with "make -j2 -O" on a slowish system to see how this can be annoying in practice. Moreover, since the output of the recipes is no longer connected to a terminal, automatic colorization of output no longer work: no more good results in green and failures in red (unless the user forces them by exporting AM_COLOR_TESTS to "always"). In defense of the '-O' option, I must say that I see how such an option can be actually useful to produce more readable logs when the output is being redirected to a file (so that one doesn't care about real-time feedback on the console); but for most "interactive" usages, I believe its downsides definitely overweight its advantages. So please don't ever make that option the default; if you really really want to, at least put in place some smart checks that only enable '-O' when the output is *not* a tty, so that we can have the best of both worlds (useful feedback for interactive runs, more readable logs for batch runs). Thanks, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
105 posts
|
I'm guessing here but I imagine the main problem comes with delaying the results of submakes?
I haven't tested to see if this is how the new feature works or not. I don't think it's completely necessary to keep all output from one submake together. so turning that off might make things more interactive, Per-target syncing is a valid compromise.
Tim
On 30 April 2013 10:48, Stefano Lattarini
<[hidden email]> wrote:
I'm sorry to say that the new -O option can interact *badly* -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
105 posts
|
What I mean is that:
might be a good interactive default rather than -Omake../make -Otarget Tim
On 30 April 2013 10:55, Tim Murphy
<[hidden email]> wrote:
-- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
In reply to
this post by Tim Murphy-4
On 04/30/2013 11:55 AM, Tim Murphy wrote:
> I'm guessing here but I imagine the main problem comes with delaying the > results of submakes? > I think so, yes. The Distributed Make from Sun implements a similar output synchronization feature, and doesn't have the output delay problems I have experienced with GNU make (while it shares the issues with automatic output colorization). > I haven't tested to see if this is how the new feature works or not. I > don't think it's completely necessary to keep all output from one submake > together. so turning that off might make things more interactive, > Per-target syncing is a valid compromise. > Indeed. > Regards, > > Tim I leave my original message quoted here in tis entirety, for reference:
... [
show rest of quote]
_______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
In reply to
this post by Tim Murphy-4
On 04/30/2013 12:01 PM, Tim Murphy wrote:
> What I mean is that: > > ./make -Otarget > > might be a good interactive default rather than -Omake. > I wasn't even aware of those differences; as of latest Git commit 'moved-to-git-46-g19a69ba', I don't see them documented in either the help screen, the manpage, the texinfo manual, nor the NEWS file. > Colouring is another issue which I would imagine could be done another way > to let us have the best of both worlds. > That is not trivial to do I think. For example, Automake-generated testsuites check whether the stdout is a tty to decide whether or not to automatically enable output colorization. And testsuites produced by Autotest do the same, AFAIK. If the make connects the stdout of those processes to non-tty files behind the scene, those checks are doomed to fail. Regards, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
105 posts
|
I know this isn't going to go down all that well, but I really think the output should be annotated in such a way that colourisation could be applied to the log file after a build has already finished.
e..g you load a makefile into VIM - it can colourise it. Or a bit of C source code. Why not the log of a build you did yesteday? It's still very nice to be able to distinguish things by colour later on.
On 30 April 2013 11:16, Stefano Lattarini
<[hidden email]> wrote:
-- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
On 04/30/2013 01:22 PM, Tim Murphy wrote:
> I know this isn't going to go down all that well, but I really think the > output should be annotated in such a way that colourisation could be > applied to the log file after a build has already finished. > While this might (underline "might" ;-) be an interesting feature, it is orthogonal to the issue under question -- that is, the need to retain the ability to automatically determine whether the output is going to a tty or a "log file" (that is, any file that is not a tty). > e..g you load a makefile into VIM - it can colourise it. Or a bit of C > source code. Why not the log of a build you did yesteday? It's still very > nice to be able to distinguish things by colour later on. > And notice that, currently, such a post-run colorization is easy to obtain, if you are only interested in having it working 99% of the time: just color a leading "PASS:" string in green, a leading "FAIL:" in red, e leading "SKIP:" in blue, etc. Regards, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
2247 posts
|
In reply to
this post by Stefano Lattarini
Just to be clear, you're saying that the testsuite runs as one long
operation, updating one target, and the recipe invokes one test script, right? I can see how that environment might be problematic for this new feature. It works much better for lots of smaller targets. However, you could avoid this issue if you specify that the target to be built is a sub-make, by prefixing it with a "+" operator for example. If this is true and you use -Otarget or -Ojob then make will not capture the output from that job. Of course this also has the unfortunate side-effect that running "make -n" will still try to run the test scripts. Hm. On Tue, 2013-04-30 at 11:48 +0200, Stefano Lattarini wrote: > So please don't ever make that option the default; if you really > really want to, at least put in place some smart checks that only > enable '-O' when the output is *not* a tty, so that we can have the > best of both worlds (useful feedback for interactive runs, more > readable logs for batch runs). This is a possibility for sure. I have to say that my experience with parallel builds hasn't been as wonderful as others here. I often get output which is corrupted, and not just by intermixing whole lines but also by having individual lines intermixed (that is the beginning of the line is one job's output, then the end of the line is another job's output, etc.) This kind of corruption is often completely unrecoverable and I simply re-run the build without parallelism enabled. I think it depends on how much output jobs emit and how many you are running at the same time. It could also be that Windows is better about avoiding interrupted writes to the same device. Tim Murphy < [hidden email]> writes: > What I mean is that: > ./make -Otarget > might be a good interactive default rather than -Omake. I always intended that and never suggested -Omake would be the default. I think -Omake is only really useful for completely automated, background builds. It wouldn't ever be something someone would use if they wanted to watch the build interactively. > I haven't tested to see if this is how the new feature works or not. I > don't think it's completely necessary to keep all output from one > submake together. so turning that off might make things more > interactive, Per-target syncing is a valid compromise. This is the default. If you use -Otarget or (-Ojob) then what is supposed to happen is that make uses the same sub-make detection algorithms used for jobserver, etc. and if it determines that a job it's going to run is a submake it does NOT collect its output. However, I have suspicions that this is not working properly. I have a make-based cross-compilation environment (building gcc + tools) I've been using for years, and I tried it with the latest make and -O and saw similar problems (sub-make output collected into one large log instead of per-target). Thinking about it right now I think I might know what the problem is, actually. I'll look at this later. Stefano Lattarini < [hidden email]> writes: > I wasn't even aware of those differences; as of latest Git commit > 'moved-to-git-46-g19a69ba', I don't see them documented in either > the help screen, the manpage, the texinfo manual, nor the NEWS file. I don't see where that code comes from? There is no Git commit in the standard GNU make archive with a SHA g19a69ba. The current HEAD on master is: 19a69ba Support dynamic object loading on MS-Windows. At any rate, the new option and option arguments are documented in the manual and there is an overview including the option arguments in the man page. The NEWS file doesn't discuss the individual option arguments, only the option. Ditto the --help output. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
Hi Paul.
On 04/30/2013 03:37 PM, Paul Smith wrote: > Just to be clear, you're saying that the testsuite runs as one long > operation, updating one target, and the recipe invokes one test script, > right? > No; the testsuite runs as a recursive make invocation (yes, this is sadly truly needed in order to support all the features offered by the Automake parallel testsuite harness --- sorry), but each test script (and there can be hundreds of them, as is the case for GNU coreutils or GNU automake itself) is run as a separate target, explicit for tests which have no extension and pattern-based for tests that have an extension. > I can see how that environment might be problematic for this new > feature. It works much better for lots of smaller targets. > See above: we are speaking of hundreds of little targets here ... > However, you could avoid this issue if you specify that the target to be > built is a sub-make, > The problem here is exactly the fact that the output of the sub-make is not displayed until its execution has ended. > by prefixing it with a "+" operator for example. > BTW, the "+" operator is not portable soem other make implementations still targeted by Automake (e.g., Solaris make). > If this is true and you use -Otarget or -Ojob then make will not capture > the output from that job. > > Of course this also has the unfortunate side-effect that running "make > -n" will still try to run the test scripts. > This would be horrific of course ;-)
... [
show rest of quote]
OTOH, mine has been very good, and I almost alway use "make -j8" or higher
when building GNU software (or other good software the relies on GNU make, like Git). The only real issues I've experienced so far have been when I used the '-O' option ... > I often get output which is corrupted, and > not just by intermixing whole lines but also by having individual lines > intermixed (that is the beginning of the line is one job's output, then > the end of the line is another job's output, etc.) > Most of these issues are solved using silent-rules, as offered by Automake or implemented ad-hoc by build systems of other software (Git, Linux). I think the best ways to use parallel GNU make are: - with silent rules when the output goes to a tty (it makes the output visually scannable, and makes warnings and error messages form compilers and most other tools stand out much more clearly). - with verbose rules and possibly the -O option enabled when the output goes to a log, to simplify post-build debugging and analysis (especially useful with CI systems and the like). > This kind of corruption is often completely unrecoverable and I simply > re-run the build without parallelism enabled. > > I think it depends on how much output jobs emit and how many you are > running at the same time. > Indeed; this corresponds to my experiences so far. > It could also be that Windows is better about > avoiding interrupted writes to the same device. > Not using windows, I cannot say anything here, sorry.
... [
show rest of quote]
This appears not to be the case, actually ...
> However, I have suspicions that this is not working properly. I have a > make-based cross-compilation environment (building gcc + tools) I've > been using for years, and I tried it with the latest make and -O and saw > similar problems (sub-make output collected into one large log instead > of per-target). > Glad to know you can reproduce this. > Thinking about it right now I think I might know what the problem is, > actually. I'll look at this later. > Thanks! > Stefano Lattarini < [hidden email]> writes: >> I wasn't even aware of those differences; as of latest Git commit >> 'moved-to-git-46-g19a69ba', I don't see them documented in either >> the help screen, the manpage, the texinfo manual, nor the NEWS file. > > I don't see where that code comes from? There is no Git commit in the > standard GNU make archive with a SHA g19a69ba. > The SHA is 19a69ba; the leading 'g' is added by the "git describe" command. Sorry for not saying that explicitly. > The current HEAD on master is: > > 19a69ba Support dynamic object loading on MS-Windows. > We are referring to the same commit then. > At any rate, the new option and option arguments are documented in the > manual > Then I must be messing something up locally. I'll see if I can solve this. > and there is an overview including the option arguments in the > man page. The NEWS file doesn't discuss the individual option > arguments, only the option. Ditto the --help output. > Best regards, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Tue, 2013-04-30 at 16:04 +0200, Stefano Lattarini wrote:
... [
show rest of quote]
This should work very well with -Otarget then, except for the colorization/highlighting issue... once it works as expected. I'll look into this issue later and I would be interested to see your experience with it once it's resolved. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Tue, 2013-04-30 at 10:39 -0400, Paul Smith wrote:
... [
show rest of quote]
OK, I found this bug. Definitely make recursion was not being handled properly with -Otarget and -Ojob in some situations; this broke as a side effect of my cleanup to reuse the same temporary file for the entire target, regardless of the output mode. This should be fixed now. Those who use recursive makefiles and were seeing annoying delays in output with -O, please try again with the latest commit and see if it works any better for you now. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
Hi Paul.
On 05/01/2013 02:04 PM, Paul Smith wrote:
... [
show rest of quote]
I can confirm the problem has disappeared for me. Thanks!
(The issue with extra "Entering/Leaving directory" messages is still present though; I hope it can be fixed before the release ...). Best regards, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
1236 posts
|
In reply to
this post by Paul Smith-20
... [
show rest of quote]
You forgot to make the same change in the WINDOWS32 branch. I did that in commit a87ff20. > This should be fixed now. Those who use recursive makefiles and were > seeing annoying delays in output with -O, please try again with the > latest commit and see if it works any better for you now. Unfortunately, the delays are still here. Moreover, it looks like this change introduced some kind of regression. With the following Makefile: all: simple recursive simple: one two one two: @echo start $@ @sleep 1 @echo stop $@ @-false recursive: rec1 rec2 rec1 rec2: @echo start $@ $(MAKE) -f mkfsync simple @echo stop $@ I see this output with the previous version: D:\gnu\make-3.82.90_GIT_2013-04-20>gnumake -f mkfsync -j -O gnumake -f mkfsync simple gnumake -f mkfsync simple mkfsync:6: recipe for target 'one' failed gnumake: [one] Error 1 (ignored) start one stop one mkfsync:6: recipe for target 'two' failed gnumake: [two] Error 1 (ignored) start two stop two start rec2 gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'one' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start one stop one gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'two' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [two] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start two stop two gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' stop rec2 start rec1 gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'one' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start one stop one gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' mkfsync:6: recipe for target 'two' failed gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: [two] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' start two stop two gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-04-20' stop rec1 Notice in particular how start rec1..stop rec1 occludes its sub-targets, and the same for rec2. After the change I see this: D:\gnu\make-3.82.90_GIT_2013-05-01>gnumake -f mkfsync -j -O gnumake -f mkfsync simple gnumake -f mkfsync simple gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start one stop one mkfsync:6: recipe for target 'one' failed gnumake: [one] Error 1 (ignored) start two stop two mkfsync:6: recipe for target 'two' failed gnumake: [two] Error 1 (ignored) gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start one stop one mkfsync:6: recipe for target 'one' failed gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start two stop two mkfsync:6: recipe for target 'two' failed gnumake[1]: [two] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start one stop one mkfsync:6: recipe for target 'one' failed gnumake[1]: [one] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Entering directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start two stop two mkfsync:6: recipe for target 'two' failed gnumake[1]: [two] Error 1 (ignored) gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' start rec1 stop rec1 start rec2 stop rec2 And now rec1 and rec2 are announced only at the end. I see the same result on GNU/Linux, so this isn't a Windows specific problem. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
On 05/01/2013 05:26 PM, Eli Zaretskii wrote:
> > Unfortunately, the delays are still here. > I can say they're no longer there *in my use case*; I haven't tested other use cases though. Hope this is sorted out soon... Thanks, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
In reply to
this post by Eli Zaretskii
On Wed, 2013-05-01 at 18:26 +0300, Eli Zaretskii wrote:
> You forgot to make the same change in the WINDOWS32 branch. I did > that in commit a87ff20. Sorry, I missed that. > > This should be fixed now. Those who use recursive makefiles and were > > seeing annoying delays in output with -O, please try again with the > > latest commit and see if it works any better for you now. > > Unfortunately, the delays are still here. Very odd. This is the test program I used; can you verify: recurse: ; $(MAKE) -f $(firstword $(MAKEFILE_LIST)) all all: 2 4 6 .RECIPEPREFIX := | 2 4 6 : |@ echo start $@ |@ sleep $@ |@ echo end $@ Now running: $ make -Omake --no-print-directory -j -f ... should wait for 6 seconds, then print everything at the end. Running this on the other hand: $ make -O --no-print-directory -j -f ... should show "start 2"/"end 2" after 2 seconds, "start 4"/"end 4" after 2 more seconds (4 total), etc. And, just for completeness, running this: $ make -Ojob --no-print-directory -j -f ... should show all the "start 2", "start 4", "start 6" right away (possibly out of order), then after 2 seconds "end 2", then 2 seconds later "end 4", etc. Is that what you see? If so then the feature is working as expected. I'd be interested to know more details of the makefiles where you're still seeing stuttering behavior. Are the targets in this environment printing a lot of output per recipe? > Moreover, it looks like > this change introduced some kind of regression. With the following > Makefile: I don't think this is a regression. It's unfortunate but I don't see any alternative. BTW, you might find it simpler to use --no-print-directory as above to get rid of the enter/leave stuff until I work out how to do it better. > Notice in particular how start rec1..stop rec1 occludes its > sub-targets, and the same for rec2. "Occludes"? I don't sense anything occluded here. Maybe a terminology error... did you mean something like "brackets"? Yes, that's the behavior we'd expect to see if make was not treating the sub-make properly: it saves up the sub-make output and prints it once the sub-make is completed. The goal of this fix is to change that... > After the change I see this: > gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' > gnumake[1]: Leaving directory 'D:/gnu/make-3.82.90_GIT_2013-05-01' > start rec1 > stop rec1 > start rec2 > stop rec2 > > And now rec1 and rec2 are announced only at the end. This isn't a bug, this is expected behavior. It's slightly unfortunate, I agree. Consider your rule: > rec1 rec2: > @echo start $@ > $(MAKE) -f mkfsync simple > @echo stop $@ Here we have a 3-line recipe: the first and third are not considered by make to be recursive, while the second one is (due to the presence of $(MAKE)) When we run with -O (-Otarget), make will save up the output from the entire recipe and print it at the end, EXCEPT that output from a recursive line is not saved: that's the entire point of this feature, to allow sub-makes to show their output as they go. So, the results of lines one and three (the echo's) are saved until the "rec1" target is completely built, and printed at the end, while the results of the make invocation in between are not saved, which is just what you see. If you want different behavior you can change your rule to use "+" on the two echo lines, so that they're also considered recursive and not saved up. Now this recipe is basically run in -Ojob mode, BTW. "+" has other side-effects though (running even in -n mode) which might be unpleasant. The only alternative to this would be for make to try to figure out if ANY of the lines in the recipe will be recursive, and if so treat ALL the lines as if -Ojob was enabled (show output as soon as the line is complete), while not treating them as recursive. However since today we don't parse lines completely until we're about to execute them, this would be a not-insignificant change I think. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
... [
show rest of quote]
Yes. But I thought the change was about -Otarget, not -Ojob. Stefano was complaining about a plain -O, so -Ojob is not what was his problem. > I don't think this is a regression. It's unfortunate but I don't see > any alternative. Too bad.
... [
show rest of quote]
That is completely unexpected for me: I thought that -Otarget meant that all the output from making a target, e.g. rec1, will be output as a single unit. If that's not the intent, then why is it called "target"? _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
In reply to
this post by Paul Smith-20
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email], [hidden email] > Date: Wed, 01 May 2013 14:41:25 -0400 > > If you want different behavior you can change your rule to use "+" on > the two echo lines, so that they're also considered recursive and not > saved up. If I do that, the echo from rec1 and rec2 mix up: D:\gnu\make-3.82.90_GIT_2013-05-01>gnumake -f mkfsync3 -j -O start rec2start <<<<<<<<<<<<<<<<<<<< rec1 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< gnumake -f mkfsync simple gnumake -f mkfsync simple Is this also expected? Anyway, my mental model of what's going on under -O is now completely shattered. I cannot understand what good is this behavior: > entire recipe and print it at the end, EXCEPT that output from a > recursive line is not saved: that's the entire point of this feature, to > allow sub-makes to show their output as they go. Why is it important to make that exception? And shouldn't we have an option _not_ to make such an exception, but without -Omake, i.e. without waiting for the whole session to end? Whenever any top-level recipe finishes, it is flushed to the screen as a single unit. Does this make sense? _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
In reply to
this post by Eli Zaretskii
On Wed, 2013-05-01 at 22:08 +0300, Eli Zaretskii wrote:
> Yes. But I thought the change was about -Otarget, not -Ojob. Stefano > was complaining about a plain -O, so -Ojob is not what was his > problem. Yes, it is about -Otarget. As I said, I added -Ojob output "just for completeness". The important distinctions (for this thread) are between -Otarget and -Omake. If you see the behavior I describe for those two flags then things are working as expected. If you still see choppiness in the output with -Otarget and you see the behavior I describe, I'd be interested to know more about the targets you're building. I'd also be interested to know if using -Otarget vs. -Ojob makes any difference in the choppiness. If your recipe normally runs for 5 seconds (say) and it continually generates output during that time, then yes, certainly the -O feature will result in choppiness because instead of a sequence of continuous output over 5 seconds you get 5 seconds of silence, followed by all the output. That is the nature of the beast... if this bothers you (more than having interleaved parallel output) best to not use -O. I suggest that for MOST makefile targets, this is not the normal behavior. Most makefile targets (building a .o for example) are built relatively quickly and, more importantly, they don't expect to see much output except on failure. I would also point out that the more output your targets generate the more likely you are to get output corruption when running with high -j, and the more you'd be likely to benefit from -O. > That is completely unexpected for me: I thought that -Otarget meant > that all the output from making a target, e.g. rec1, will be output as > a single unit. If that's not the intent, then why is it called > "target"? That IS the intent. However, in the presence of makefile recursion this model fails. Consider if you have a makefile, like every single automake makefile for example!, where the "top-level" target is nothing more than a recursive invocation of a sub-make with some extra arguments, and the sub-make actually does all the work: all: config.h $(MAKE) $(AM_MAKEFLAGS) all-recursive Now, if you do nothing special for recursive make, you'll get no output from the entire build until it is completely done, because all the output from the recursive make command is going to the temporary file for that target, then it all gets dumped at the same time. I think this makes the output sync feature much less useful: it's only appropriate for building in the background where no one looks at the output until it's done. If you want that behavior, though, that's exactly what the -Omake option selects so it's available. For -Otarget we introduce an exception to deal with this problem: if we detect we're about to invoke a recursive make, we do NOT redirect the output to the temp file. We let each target of that make print its output immediately. This way you don't have to wait for an entire recursive make to finish before you see any of its output. > Why is it important to make that exception? And shouldn't we have an > option _not_ to make such an exception, but without -Omake, > i.e. without waiting for the whole session to end? Whenever any > top-level recipe finishes, it is flushed to the screen as a single > unit. Does this make sense? I don't understand the change that you're suggesting. That's exactly what -Omake does today: whenever any recipe finishes it is flushed to the screen as a single unit, and no special handling is given to recursive makes. If we can improve on this I'm very interested to hear the details. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email], [hidden email] > Date: Wed, 01 May 2013 16:27:58 -0400 > > If your recipe normally runs for 5 seconds (say) and it continually > generates output during that time, then yes, certainly the -O feature > will result in choppiness because instead of a sequence of continuous > output over 5 seconds you get 5 seconds of silence, followed by all the > output. I think that's the reason, yes.
... [
show rest of quote]
Not every Makefile looks like that on its top level. I agree that we should cater to the above, but perhaps we could do that without "punishing" the other use cases. For example, perhaps we should have a -Osub-make option that will _not_ activate sync-output on the top level, only in the sub-make's. This should produce the desired results, I think. > > shouldn't we have an option _not_ to make such an exception, but > > without -Omake, i.e. without waiting for the whole session to end? > > Whenever any top-level recipe finishes, it is flushed to the > > screen as a single unit. Does this make sense? > > I don't understand the change that you're suggesting. That's exactly > what -Omake does today: whenever any recipe finishes it is flushed to > the screen as a single unit, and no special handling is given to > recursive makes. In my case, I see all the output at once. Maybe I misunderstand what -Omake is supposed to do, too. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
105 posts
|
One optimisation I have thought of in the past for this situation would be to allow a single "job" to hold onto the lock when it obtained it.
This way it could output directly to the console while all other jobs would have to buffer. When it released, the next job lucky enough to grab the lock might have a full buffer already.
It might appear to be less choppy. Not sure how it would perform.
Regards,
Tim
On 2 May 2013 03:53, Eli Zaretskii
<[hidden email]> wrote:
> Date: Wed, 01 May 2013 16:27:58 -0400 -- You could help some brave and decent people to have access to uncensored news by making a donation at: http://www.thezimbabwean.co.uk/friends/ _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
«
Return to Gnu - Make - Bugs |
44 views
Some serious issues with the new -O option
| Classic | | List | | Threaded |
123
123
|
2247 posts
|
Eli Zaretskii <
[hidden email]> writes:
... [
show rest of quote]
You're right, I'm wrong. Using "+" is not really like -Ojob, because with -Ojob we still sync the output of each line. Using "+" the job just sends the output directly to the terminal with no sync. Thus this mixing up is what I'd expect (same as not using -O at all). I begin to wonder if this really requires a new per-line special prefix character like "@", "+", "-", that controls the syncing of that particular line. I'm very reluctant to go there as it is a BIG change and a backward-compat issue. Also it seems that unlike the existing prefix characters, for this one we'd have as much need for a way to turn OFF sync as to turn it ON... bleah. On Thu, 2013-05-02 at 05:53 +0300, Eli Zaretskii wrote:
... [
show rest of quote]
Can you clarify what the "desired results" are? I seem to have lost the thread. What is the behavior you see now that you are trying to avoid (or you don't see that you're trying to achieve)? Capture of the sub-make will mean that the entire output of that sub-make, and all of its recipes including ITS sub-sub-makes, will be sent to a temporary file and not displayed on the screen until the entire sub-make is completed. In what situation would we want to choose this, regardless of level of sub-make? In general I see no benefit in trying to special-case any particular level of make. For some builds the top level does all the work. For some the second level. For some the third. For many, different levels for different parts of the same build. > > I don't understand the change that you're suggesting. That's exactly > > what -Omake does today: whenever any recipe finishes it is flushed to > > the screen as a single unit, and no special handling is given to > > recursive makes. > > In my case, I see all the output at once. Maybe I misunderstand what > -Omake is supposed to do, too. I think you and I said the same thing: the output from recursive makes is saved up and flushed all at once...? Tim Murphy < [hidden email]> writes: > One optimisation I have thought of in the past for this situation > would be to allow a single "job" to hold onto the lock when it > obtained it. > > This way it could output directly to the console while all other jobs > would have to buffer. When it released, the next job lucky enough to > grab the lock might have a full buffer already. > > It might appear to be less choppy. Not sure how it would perform. It might be less choppy (or it might not: it depends on your targets: are they all more-or-less equally chatty?) but we discussed this possibility and decided that it would be too costly in terms of performance. All you need is for one long-running job to get the terminal and pretty soon it's the only job running as all others have finished but can't continue with the next one until they can grab the terminal and dump their output. Personally I think this is a serious enough problem that it's probably not even worth doing the work in GNU make to provide this as an option. I suspect people would probably much rather just not use -O and live with interleaved output, or use -O and live with some choppy output, than suffer essentially random increases in build times. I could be wrong though. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
... [
show rest of quote]
The "desired results" in my original use case are that the output of remaking each target is shown as one chunk in the order in which it is expected, i.e. from the first command to the last. "Remaking a target", a.k.a. "recipe" in this context are all the commands that rebuild a single target. E.g., in this snippet: foo: bar cmd1 cmd2 $(MAKE) -C foo something-else cmd3 all the 4 commands and whatever they emit should be output in one go, and in the order they are in the Makefile. The "desired results" in the example you showed, i.e. > all: config.h > $(MAKE) $(AM_MAKEFLAGS) all-recursive are that -O is in effect only for the sub-make which runs all-recursive. > Capture of the sub-make will mean that the entire output of that > sub-make, and all of its recipes including ITS sub-sub-makes, will be > sent to a temporary file and not displayed on the screen until the > entire sub-make is completed. In what situation would we want to choose > this, regardless of level of sub-make? Not "regardless of level", of course. If there are many levels of recursion, then doing that is not what most people will want. But if "make all-recursive" does not recurse anymore, then yes, what I described will be good. > In general I see no benefit in trying to special-case any particular > level of make. For some builds the top level does all the work. For > some the second level. For some the third. For many, different levels > for different parts of the same build. The user always knows what she is going to run, or at least ought to know. I think we already established that blindly appending -O to the Make command might cause surprising and even disappointing or annoying results, if one does it for a use case that does not play well with this feature. So some degree of adaptation between -O and its sub-modes to the depth and "breadth" (in parallelism sense) of the build is necessary anyway.
... [
show rest of quote]
No, that's not what I said. I said whenever a _recipe_ finishes, not whenever the entire Make run finishes. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Thu, 2013-05-02 at 20:24 +0300, Eli Zaretskii wrote:
... [
show rest of quote]
This is the way -Omake works. If you want that, you should set -Omake. > The "desired results" in the example you showed, i.e. > > > all: config.h > > $(MAKE) $(AM_MAKEFLAGS) all-recursive > > are that -O is in effect only for the sub-make which runs > all-recursive. I'm not sure what you mean by "ONLY for the sub-make" but if I understand correctly this is the way -Otarget works. So, you can have either method. Maybe what you're suggesting is that you want BOTH methods, but for different targets, in the same build? Today we don't support that. This is what I was talking about when I referred to a new prefix character on a per line basis to turn on/off ordered mode. I think trying to control this through the command line, by choosing different MAKELEVEL values to behave differently, is not a good solution. However I don't want to go there yet. It's a big, disruptive change and I'm not convinced that (a) we can't do something to obviate it and (b) even if we can't that the use-case is important enough to justify it. The build still behaves just as it did before, it's just that in your original case the output is shown in a strange order.
... [
show rest of quote]
I don't think I agree with much of the above. But it's a matter of opinion so we'll just have to wait and see.
... [
show rest of quote]
-Omake only has any relevance when doing recursive makes. If you run "make -Otarget" and "make -Omake" in a non-recursive make environment, you will get identical behavior. The one and only difference between them is that when running a recursive make, -Otarget WILL NOT capture the output of the sub-make, leaving whatever it prints going to the same stdout as the parent make, and -Omake WILL capture the output of the sub-make in the temporary file, to be dumped after the recipe is complete. There's no -O mode which will save up the entire output of a non-recursive make and dump it all at the end. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email], [hidden email] > Date: Thu, 02 May 2013 16:21:36 -0400 > > The one and only difference between them is that when running a > recursive make, -Otarget WILL NOT capture the output of the sub-make, > leaving whatever it prints going to the same stdout as the parent make, > and -Omake WILL capture the output of the sub-make in the temporary > file, to be dumped after the recipe is complete. Thanks for explaining that. I will have to try a few things to make sure I really get it this time, but one thing I can already say is that 'target' and 'make' are not very good names for these modes, since their semantics is quite different from the literal meaning of these two words. That difference creates a semantic dissonance that we should try to avoid, I think. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Fri, 2013-05-03 at 09:50 +0300, Eli Zaretskii wrote:
... [
show rest of quote]
It's good that we're having this discussion: I want to use it to try to inform my editing of the GNU make manual to be sure it's as clear as possible. I don't think the names are so inaccurate. I don't want to name things based solely on the details of how they differ from one another, which is all I was describing above. I prefer to name them based on how their most salient behavior manifests to the user. The way the user experiences the -Ojob option's results is that the output of every line of each recipe is dumped as soon as that line is complete. The way the user experiences the -Otarget option's results is that the output of all the lines of a given target are dumped at the same time once the target is completely built. The way the user experiences the -Omake option's results is that the output of an entire recursive make invocation is dumped, together, once the recursive make has completed. The issue of how -Otarget handles recursive make is, IMO, a detail necessitated by the architecture of recursive make invocations. I don't know that it's feasible to reflect that detail in the name. On the other hand I'm certainly not married to the current terms and I'm quite happy to change them if better ones can be found. It has already been suggested that -Oline would be better than -Ojob, and -Orecipe would be better than -Otarget, and -Omakefile would be better than -Omake. The current names are based more around _actions_ while the new suggestions are based more around semantic elements of make. To me -Omake is the most problematic. -Omakefile is not much better; in fact it might be worse (after all you can and often do invoke a recursive make on the same makefile). It would be nice to be more clear about the fact it applies only to recursive make invocations. Something like -Osubmake might be more accurate, except that I don't think we use the term "sub-make" in the documentation: we use "recursive make". Is -Orecursive better? _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email], [hidden email] > Date: Fri, 03 May 2013 07:47:09 -0400 > > The way the user experiences the -Ojob option's results is that the > output of every line of each recipe is dumped as soon as that line is > complete. I would suggest -Oline or -Ocommand for this. "Job" is not necessarily recognizable by users of Make for what we mean when we talk about that. > The issue of how -Otarget handles recursive make is, IMO, a detail > necessitated by the architecture of recursive make invocations. I don't > know that it's feasible to reflect that detail in the name. It is a detail that IMO significantly qualifies the "target" part. In particular, targets that include little or nothing except a recursive invocations will be entirely exempt from this "target" scope. > To me -Omake is the most problematic. -Omakefile is not much better; in > fact it might be worse (after all you can and often do invoke a > recursive make on the same makefile). It would be nice to be more clear > about the fact it applies only to recursive make invocations. Something > like -Osubmake might be more accurate, except that I don't think we use > the term "sub-make" in the documentation: we use "recursive make". Is > -Orecursive better? Yes, I think -Orecursive is better. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Fri, 2013-05-03 at 15:22 +0300, Eli Zaretskii wrote:
> > The issue of how -Otarget handles recursive make is, IMO, a detail > > necessitated by the architecture of recursive make invocations. I don't > > know that it's feasible to reflect that detail in the name. > > It is a detail that IMO significantly qualifies the "target" part. In > particular, targets that include little or nothing except a recursive > invocations will be entirely exempt from this "target" scope. Personally I think it would be MORE confusing if you requested -Otarget and the output for many or even all the targets in your build was not printed until the build finished. You're concentrating on the one recursive make target and saying "this doesn't follow the rule", while I'm concentrating on all targets in the sub-make and saying "let's make sure all of these follow the rule" (that their output is shown as soon as that target is complete). Recursive make targets are merely artifacts of the build. Users don't care about them; they're just used by makefile authors to organize things. If the makefile author rewrote the makefiles to be non-recursive, users wouldn't notice at all. Anyway that's how I look at it. Anyway. I'm happy to entertain naming suggestions that try to capture this exceptional treatment of recursive make but I have no ideas myself. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
... [
show rest of quote]
Sorry, I don't understand this way of reasoning at all. Maybe we are talking about different things. Let me try again. Consider a target like this: foo: .... prepare something $(MAKE) -C subdir doit touch foo-done Is this a legitimate and good use of recursive Make? Should -O cater to this use case? Or is this use case rare (or even bad practice), so much so that the _default_ mode of -O is allowed to confuse the heck out of the user, who will see the output of the subdir job _before_ she sees the output of preparations for that subdir job? If the reason for exempting recursive Make invocations from -Otarget is that you want it to cover a different use case, like this: foo: ... $(MAKE) -C subdir doit then how about if this exemption will only be applied if the recipe has a single command? If the single-command-in-recursive-invocation is _not_ the use case which -Otarget is optimized for, then what use case is? IOW, what is the "ideal" Makefile where -Otarget doesn't have any downsides whatsoever? _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
I have a solution for this problem that I think will work well, and will
be simple to implement. Suppose we do this: if we're about to invoke a line marked recursive and we're in -Otarget mode, then before we run it we'll show the current contents of the temp file (using the normal synchronized output function). This will ensure that output from lines before the recursive make will be shown before the targets in the recursive make. It's not 100% identical but I can't see any way to do better. Thoughts? On Fri, 2013-05-03 at 16:39 +0300, Eli Zaretskii wrote: > then how about if this exemption will only be applied if the recipe > has a single command? In this case, if a recipe consisted of only one line then every target in the submake will be output immediately when it's finished, but as soon as you add another line to the recipe, like "@echo "Build is done!", now all of a sudden you get no output from the entire sub-make until the end. That would be too confusing. > If the single-command-in-recursive-invocation is _not_ the use case > which -Otarget is optimized for, then what use case is? -Otarget is not really about recursive invocations at all. It's there for the non-recursive targets. It would be nice if it worked well for the recursive targets, too, obviously. Consider every target in the entirety of build, including all submakes and all their targets as well, as one long list of targets (for example the results of a serial build). The fraction of those targets that are invoking sub-makes will be, at most, very small. -Otarget wants to collect the output of each individual target, then show the output for each target as soon as that target finishes. That's what users (should) expect when using this option. In the case of recursive make targets, this presents an unsolveable contradiction: how can you both show the output of EVERY target (including the targets run by the submake) as soon as it completes, and still not show the output of any target (including the targets that invoke a submake) before it's complete? You can't do both! The -Omake option chooses the latter as more important: it will delay the output until the submake is complete. The -Otarget option chooses the former as more important: it wants to behave properly for the large majority of targets which are not invoking a recursive make. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email], [hidden email] > Date: Fri, 03 May 2013 12:55:26 -0400 > > Suppose we do this: if we're about to invoke a line marked recursive and > we're in -Otarget mode, then before we run it we'll show the current > contents of the temp file (using the normal synchronized output > function). Sounds good, I think we should try that. > This will ensure that output from lines before the recursive make will > be shown before the targets in the recursive make. It's not 100% > identical but I can't see any way to do better. Why isn't it identical? > Consider every target in the entirety of build, including all submakes > and all their targets as well, as one long list of targets (for example > the results of a serial build). The fraction of those targets that are > invoking sub-makes will be, at most, very small. That depends. I think you will find a different fraction in Emacs, especially when it bootstraps itself (which involves byte-compiling all the Lisp files in a 3-level deep recursion, the last of which passes to Make a very long list of files which all can be compiled in parallel; see lisp/Makefile.in in Emacs). > -Otarget wants to collect the output of each individual target, then > show the output for each target as soon as that target finishes. That's > what users (should) expect when using this option. > > In the case of recursive make targets, this presents an unsolveable > contradiction: how can you both show the output of EVERY target > (including the targets run by the submake) as soon as it completes, and > still not show the output of any target (including the targets that > invoke a submake) before it's complete? You can't do both! But I don't think there's a requirement to avoid showing incomplete output. The only requirement is not to mix output from two or more parallel jobs, that's all. I guess that brings us back to your suggestion above. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Fri, 2013-05-03 at 21:15 +0300, Eli Zaretskii wrote:
> > This will ensure that output from lines before the recursive make will > > be shown before the targets in the recursive make. It's not 100% > > identical but I can't see any way to do better. > > Why isn't it identical? It's not identical in two ways: first it's not identical to -Otarget because the output before and after the recursion are not shown in a continuous block. In: all: @echo start $(MAKE) -C subdir @echo end the "start" and "end" will have other stuff (not just the other targets in that sub-make, but ANY other targets that happen to finish during that time) between them. Obviously that's kind of the point; nevertheless it's a difference. Second, there's a difference from some ideal solution if the recursive line generates output; suppose instead of the above you wrote it like this: all: ; @echo one ; echo two ; $(MAKE) -C subdir ; echo end Since the output of the entire recursive line is not synchronized but just left to go to stdout, the output generated by the "echo" commands can (a) interrupt other synchronized output, (b) have other output interrupt it. But, it will still be shown in order! :-)
... [
show rest of quote]
Sure, but that's just one invocation of recursive make then that one recursive make builds all those targets. So that counts as one recursive make target and lots of individual bytecompile targets. Of course it's possible to write your makefile such that you run a separate recursive make command for each target, but that seems pretty wasteful of resources.
... [
show rest of quote]
That's the lowest-level requirement. But if that were the ONLY requirement we'd simply implement -Ojob and we'd be done. That's all we need to ensure that two or more parallel job outputs are not mixed: -Ojob says that the output of each line of a recipe will be printed without any interruption from any other lines running in parallel. The -Otarget option makes a stronger statement: it says that the output from all of the lines of a recipe will be printed together without any interruption from any other recipe. In the case of targets that generate output AND invoke recursive make, we can't achieve this stronger promise, at least not completely. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
... [
show rest of quote]
This last part (about ANY other targets) is not what I thought you had in mind. > > But I don't think there's a requirement to avoid showing incomplete > > output. The only requirement is not to mix output from two or more > > parallel jobs, that's all. > > That's the lowest-level requirement. But if that were the ONLY > requirement we'd simply implement -Ojob and we'd be done. That's a misunderstanding due to overloaded meaning of "job". I meant by that the entire sequence of commands that remake one target, including any of recursive invocations needed for that. So I guess we are back at square one, which is a pity. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Fri, 2013-05-03 at 23:12 +0300, Eli Zaretskii wrote:
> > the "start" and "end" will have other stuff (not just the other targets > > in that sub-make, but ANY other targets that happen to finish during > > that time) between them. > > This last part (about ANY other targets) is not what I thought you had > in mind. No. With -Otarget and -Ojob it's never the case that an entire sub-make will "own" the output lock such that no other jobs running in parallel from other sub-makes can display output. This is something someone else mentioned the other day. Doing this would seriously compromise the parallelization. Given that today people are more-or-less satisfied to have garbled output rather than slow down their parallel builds, I find it impossible to believe they'd rather have ordered output if it reduced parallelization. When running in parallel it's always been the case, and is still the case with -O, that you must consider all the targets that could possibly be started by any make (at any level of recursion) as big grab-bag of targets that could be run at any time (subject to prerequisite relationships). Recursion is not a "sequence point" in your build, when parallelization is enabled. -O in no way changes that behavior, all it does is ensure that output from any individual line or target of the recipe will not interfere with any other individual line or target. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email], [hidden email] > Date: Fri, 03 May 2013 17:17:44 -0400 > > -O in no way changes that behavior, all it does is ensure that output > from any individual line or target of the recipe will not interfere with > any other individual line or target. At the Make's internals level, -O indeed doesn't change that behavior. But, whether we want it or not, Make users perceive the order in which they see the output as the order in which things got executed. So any significant changes in the order we show the results of remaking a target will, or at least might, be interpreted as changes in behavior. While it is OK for a non-default switch to change behavior, the modified behavior must make sense, or users will complain. For the default behavior, it must make a lot of sense. Without -O, output might be messy at times, but there's never any doubt in user's mind what happened: she understands that the jobs whose output appears on the screen at the same time run in parallel. Add -O (in its current implementation), and the user now stares at a puzzle: output appears serialized, but the order of that serialization is hard to make sense of. I'm not even sure I understand completely what confuses me when I look at such output. Perhaps my mind tries to interpret that as a non-parallel run, where a prerequisite target is always remade before the target that's dependent on that prerequisite. Or maybe some other mental mechanism is at work here. All I can say is that the order in which -O shows output confuses the heck out of me, and I need to look at it for a long time, reading and re-reading its portions, before I can convince myself that everything that needed to be remade was indeed remade, and in correct order. I guess that means I will seldom if ever use -O. And if that doesn't bother anyone, then let's just leave it at that. Either I'm an odd one out, or you will hear from others who might explain this better than I could. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
In reply to
this post by Paul Smith-20
On Fri, 2013-05-03 at 12:55 -0400, Paul Smith wrote:
> Suppose we do this: if we're about to invoke a line marked recursive > and we're in -Otarget mode, then before we run it we'll show the > current contents of the temp file (using the normal synchronized > output function). I've implemented this feature and it seems to work as expected. I also implemented the fix to the duplicate output being shown in some cases. I have two open issues I want to address before calling this feature done: first, fix make's writing of the command it's going to run (for rules that don't start with "@") as that's not working right. Second, fix the enter/leave issue. It turns out that these are currently somewhat bound together so I may have to solve the second one first. Oh, and a renaming as well :-) Eli, I did some cleanup in job.c to try to make things simpler and reduce duplication. I tried to be careful but it's quite possible I did something to disrupt the Windows version again. It's up to you if you want to fix any problems now or wait until I solve the above two issues and look at it all at the same time. There will be more disruption I think. One other thing: I changed the pump function to read from a FD but write to a FILE*, because all our other uses of stdout/stderr use FILE* and it's not wise to mix them. It works fine. While I was in there I noticed the handling of the text/binary mode. I wonder if this is not quite right. It seems to me that since we're going to be writing to stdout/stderr anyway, if we're going to set the mode at all we should be setting the mode on the temporary file to match the mode on stdout/stderr, before we write to it, rather than setting the mode on stdout/stderr to match the temporary file. What do you think? _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
Hi Paul.
On 05/05/2013 12:10 AM, Paul Smith wrote:
... [
show rest of quote]
The test 'features/output-sync' now fails for me:
Test timed out after 6 seconds Error running /storage/home/stefano/src/gnu/make/tests/../make \ (expected 0; got 14): /storage/home/stefano/src/gnu/make/tests/../make \ -f work/features/output-sync.mk.1 -j --output-sync=target Caught signal 14! FAILED (4/6 passed) Can you reproduce the failure? If not, let me know which information you need to debug this, and I'll try to provide them. Regards, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Sun, 2013-05-05 at 00:44 +0200, Stefano Lattarini wrote:
... [
show rest of quote]
It didn't fail for me. However, it's possible that the 6 second timeout is just a little too short for reliability. Look at line 178 in the test/scripts/features/output-sync file. It will look like this: #MAKE#[1]: Leaving directory '#PWD#/bar'\n", 0, 6); The "6" at the end is the timeout. Try changing that to 10 just to see if it helps. If not then it's a real problem. I'll need details about your system. Also please send the contents of the work subdirectory after the failure. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
Hi Paul.
On 05/05/2013 08:19 AM, Paul Smith wrote:
... [
show rest of quote]
Sorry to add this only now, but I realized the failure is only reproducible
if I run the testsuite with "make -j", as in "make -j8 check; and even in that case, the failure is racy. With a bare "make check", things work for me as well. On the other hand, increasing the parallelism even more, other tests start to fail as well: -*-*-*- $ make check -j16 Making check in glob make[1]: Entering directory `/storage/home/stefano/src/gnu/make/glob' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/glob' Making check in config make[1]: Entering directory `/storage/home/stefano/src/gnu/make/config' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/config' Making check in po make[1]: Entering directory `/storage/home/stefano/src/gnu/make/po' make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/po' Making check in doc make[1]: Entering directory `/storage/home/stefano/src/gnu/make/doc' make[1]: Nothing to be done for `check'. make[1]: Leaving directory `/storage/home/stefano/src/gnu/make/doc' make[1]: Entering directory `/storage/home/stefano/src/gnu/make' make loadavg make[2]: Entering directory `/storage/home/stefano/src/gnu/make' make[2]: Leaving directory `/storage/home/stefano/src/gnu/make' make check-local make[2]: Entering directory `/storage/home/stefano/src/gnu/make' cd tests && perl ./run_make_tests.pl -srcdir /storage/home/stefano/src/gnu/make -make ../make The system uptime program believes the load average to be: uptime 11:01:21 up 47 min, 4 users, load average: 0.03, 0.19, 0.21 The GNU load average checking code thinks: ./loadavg 1-minute: 0.030000 5-minute: 0.190000 15-minute: 0.210000 ------------------------------------------------------------------------------ Running tests for GNU make on Linux laptop 3.2.0-4-amd64 x86_64 GNU Make 3.82.90 ------------------------------------------------------------------------------ Clearing work... Finding tests... features/archives ....................................... ok (8 passed) features/comments ....................................... ok (1 passed) features/conditionals ................................... ok (4 passed) features/default_names .................................. ok (3 passed) features/double_colon ................................... *** Test died (features/double_colon): test_driver.pl: 468 -> scripts/features/double_colon: 77: Couldn't touch foo: Is a directory FAILED (4/4 passed) features/echoing ........................................ ok (4 passed) features/errors ......................................... *** Test died (features/errors): test_driver.pl: 468 -> scripts/features/errors: 41: Couldn't touch foo: Is a directory FAILED (0/0 passed) features/escape ......................................... FAILED (7/8 passed) features/export ......................................... ok (12 passed) features/include ........................................ Error running /storage/home/stefano/src/gnu/make/tests/../make (expected 0; got 512): /storage/home/stefano/src/gnu/make/tests/../make -f work/features/include.mk.4 Error running /storage/home/stefano/src/gnu/make/tests/../make (expected 0; got 512): /storage/home/stefano/src/gnu/make/tests/../make -f work/features/include.mk.5 FAILED (5/10 passed) features/load ........................................... ok (6 passed) features/loadapi ........................................ ok (3 passed) features/mult_rules ..................................... ok (2 passed) features/mult_targets ................................... ok (2 passed) features/order_only ..................................... FAILED (6/10 passed) features/output-sync .................................... ok (6 passed) features/override ....................................... ok (4 passed) features/parallelism .................................... ok (10 passed) features/patspecific_vars ............................... ok (10 passed) features/patternrules ................................... ok (10 passed) features/quoting ........................................ ok (1 passed) features/recursion ...................................... ok (2 passed) features/reinvoke ....................................... ok (5 passed) features/se_explicit .................................... ok (10 passed) features/se_implicit .................................... ok (9 passed) features/se_statpat ..................................... ok (4 passed) features/shell_assignment ............................... ok (4 passed) features/statipattrules ................................. ok (8 passed) features/targetvars ..................................... ok (25 passed) features/utf8 ........................................... ok (1 passed) features/varnesting ..................................... ok (2 passed) features/vpath .......................................... ok (2 passed) features/vpath2 ......................................... ok (1 passed) features/vpath3 ......................................... ok (1 passed) features/vpathgpath ..................................... ok (1 passed) features/vpathplus ...................................... ok (4 passed) functions/abspath ....................................... ok (1 passed) functions/addprefix ..................................... ok (1 passed) functions/addsuffix ..................................... ok (2 passed) functions/andor ......................................... ok (2 passed) functions/basename ...................................... ok (1 passed) functions/call .......................................... ok (3 passed) functions/dir ........................................... ok (1 passed) functions/error ......................................... ok (5 passed) functions/eval .......................................... ok (9 passed) functions/file .......................................... ok (6 passed) functions/filter-out .................................... ok (5 passed) functions/findstring .................................... ok (1 passed) functions/flavor ........................................ ok (1 passed) functions/foreach ....................................... ok (4 passed) functions/guile ......................................... N/A functions/if ............................................ ok (1 passed) functions/join .......................................... ok (1 passed) functions/notdir ........................................ ok (1 passed) functions/origin ........................................ ok (1 passed) functions/realpath ...................................... ok (2 passed) functions/shell ......................................... ok (2 passed) functions/sort .......................................... ok (2 passed) functions/strip ......................................... ok (2 passed) functions/substitution .................................. ok (3 passed) functions/suffix ........................................ ok (1 passed) functions/value ......................................... ok (1 passed) functions/warning ....................................... ok (4 passed) functions/wildcard ...................................... ok (6 passed) functions/word .......................................... ok (16 passed) misc/bs-nl .............................................. ok (18 passed) misc/close_stdout ....................................... ok (1 passed) misc/general1 ........................................... ok (1 passed) misc/general2 ........................................... ok (1 passed) misc/general3 ........................................... ok (10 passed) misc/general4 ........................................... ok (6 passed) options/dash-B .......................................... ok (8 passed) options/dash-C .......................................... ok (2 passed) options/dash-I .......................................... ok (3 passed) options/dash-W .......................................... ok (10 passed) options/dash-e .......................................... ok (1 passed) options/dash-f .......................................... ok (4 passed) options/dash-k .......................................... ok (3 passed) options/dash-l .......................................... ok (1 passed) options/dash-n .......................................... ok (6 passed) options/dash-q .......................................... ok (8 passed) options/dash-t .......................................... ok (2 passed) options/eval ............................................ ok (2 passed) options/general ......................................... ok (1 passed) options/symlinks ........................................ ok (10 passed) options/warn-undefined-variables ........................ ok (2 passed) targets/DEFAULT ......................................... ok (1 passed) targets/FORCE ........................................... ok (1 passed) targets/INTERMEDIATE .................................... ok (8 passed) targets/ONESHELL ........................................ ok (6 passed) targets/PHONY ........................................... ok (1 passed) targets/POSIX ........................................... ok (4 passed) targets/SECONDARY ....................................... ok (12 passed) targets/SILENT .......................................... ok (1 passed) targets/clean ........................................... ok (2 passed) variables/CURDIR ........................................ ok (1 passed) variables/DEFAULT_GOAL .................................. ok (5 passed) variables/INCLUDE_DIRS .................................. ok (2 passed) variables/LIBPATTERNS ................................... ok (2 passed) variables/MAKE .......................................... ok (1 passed) variables/MAKECMDGOALS .................................. ok (3 passed) variables/MAKEFILES ..................................... ok (2 passed) variables/MAKEFLAGS ..................................... ok (2 passed) variables/MAKELEVEL ..................................... ok (1 passed) variables/MAKE_RESTARTS ................................. ok (3 passed) variables/MFILE_LIST .................................... ok (1 passed) variables/SHELL ......................................... ok (8 passed) variables/automatic ..................................... ok (7 passed) variables/define ........................................ ok (16 passed) variables/flavors ....................................... ok (8 passed) variables/negative ...................................... ok (4 passed) variables/private ....................................... ok (10 passed) variables/special ....................................... ok (2 passed) variables/undefine ...................................... ok (4 passed) 10 Tests in 5 Categories Failed (See .diff files in work dir for details) :-( make[2]: *** [check-regression] Error 1 make[2]: Leaving directory `/storage/home/stefano/src/gnu/make' make[1]: *** [check-am] Error 2 make[1]: Leaving directory `/storage/home/stefano/src/gnu/make' make: *** [check-recursive] Error 1 -*-*-*- The failures are racy in this case as well. Maybe some environment variable is leaking from the 'make' process that drives the testsuite into the 'make' invocations issued by the test scripts?
... [
show rest of quote]
Regards, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
2247 posts
|
On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote:
> Sorry to add this only now, but I realized the failure is only > reproducible if I run the testsuite with "make -j", as in "make -j8 > check; and even in that case, the failure is racy. With a bare "make > check", things work for me as well. On the other hand, increasing the > parallelism even more, other tests start to fail as well: The test suite definitely cannot be run in parallel. However this should not happen (and does not happen in my environment when I run the commands above) because the test harness cleans out the environment, which will remove any of the MAKEFLAGS or MFLAGS variables that might tell the make to run in parallel when it's not expected. Can you examine your shell configuration files etc. to see if they're setting MAKEFLAGS or MFLAGS? Although if that's true then the tests should fail all the time. Can you verify that there don't seem to be any leftover test files in the tests directory? Sometimes if something doesn't get cleaned up correctly that can cause future builds to fail. However if that were the case then "make check" without -j would fail as well. I don't have an explanation for this. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
In reply to
this post by Paul Smith-20
> From: Paul Smith <
[hidden email]>
> Cc: [hidden email] > Date: Sat, 04 May 2013 17:51:05 -0400 > > Eli, I did some cleanup in job.c to try to make things simpler and > reduce duplication. I tried to be careful but it's quite possible I did > something to disrupt the Windows version again. It's up to you if you > want to fix any problems now or wait until I solve the above two issues > and look at it all at the same time. There's nothing to fix, as things still seem to work. Thanks! However, I wonder what was the reason for splitting the definition of GMK_EXPORT in two, and putting each part in a different file. The way they were together before is how programmers are used to see this stuff on Windows; splitting them will not something people will expect. I'm sure you can see this in quite a few packages out there. Here's a random example from GMP's gmp.h, slightly edited for clarity: #define __GMP_DECLSPEC_EXPORT __declspec(__dllexport__) #define __GMP_DECLSPEC_IMPORT __declspec(__dllimport__) #if __GMP_LIBGMP_DLL #if __GMP_WITHIN_GMP /* compiling to go into a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_EXPORT #else /* compiling to go into an application which will link to a DLL libgmp */ #define __GMP_DECLSPEC __GMP_DECLSPEC_IMPORT #endif #else /* all other cases */ #define __GMP_DECLSPEC #endif If you didn't like the symbol "MAIN", then we can use any other symbol, like BUILDING_GMAKE or whatever. But having this split in two is not something I'd recommend. I think it's bad for maintenance, if nothing else. > There will be more disruption I think. Looking forward to it ;-)
... [
show rest of quote]
Make never changes the I/O mode of its standard streams. And it shouldn't, since everything Make itself writes or reads is always plain text. Since the standard streams always start in text mode, your suggestion boils down to make input from the temporary file be always in text mode. That wouldn't be right. The issue here is that Make has no idea what mode will be used by its children. We redirect the children's standard streams to a file, but we cannot force them to use this or that mode (nor should we, because that is up to the child program). Since we have no idea, and we must copy to our stdout/stderr everything the child wrote, we must assume the worst. And the worst is that the child did use binary mode, and as result wrote there some bytes that cannot be read in text mode without corrupting child's output. Examples include the ^Z byte, which stops text-mode reads and writes, and lone CR characters that get stripped by text-mode reads to disappear without a trace. So we use binary mode to read from the temporary file, because that's the only way to guarantee that everything the child wrote will be read verbatim. We then need to write that stuff to our output in the same mode, to produce identical output, as if the intermediate temporary file never happened. Does this make sense? If you want a practical example to illustrate this conundrum, just ask. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
| Powered byNabble | Edit this page |
Some serious issues with the new -O option
| Classic | | List | | Threaded |
123
123
|
2247 posts
|
On Sun, 2013-05-05 at 19:36 +0300, Eli Zaretskii wrote:
> However, I wonder what was the reason for splitting the definition of > GMK_EXPORT in two, and putting each part in a different file. Well, because the gnumake.h file is intended to be installed into the user's /usr/include or similar, and included in a binary package build of make such as RPM or DEB or whatever, and be included by the user's code, and when it's included there it should NEVER (IIUC) be using the in-make variant. I wanted to separate that in-make variant out so that users would never see it or have the possibility of accidentally using it, so I moved it into our internal headers which are never installed anywhere outside our source tree and would not be included in a GNU make binary package. This is slightly more potential maintenance on our part internally but is much safer for the user which is a tradeoff I'll almost always choose. However, if you really want it back the way it was please do choose a more unique name than "MAIN". Something prefixed with "GMK_" at least.
... [
show rest of quote]
Well, I assumed that something that invoked make could set the mode and then make could inherit that mode. I don't know if that's how it works or not in Windows. And of course I doubt anyone does that. I understand your point. I just wonder if this difference might end up being visible to the user in some way. But, we'll leave it as-is. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
... [
show rest of quote]
That's true, but it's common to have the same header be included when building the application or library which provides the exported services. > However, if you really want it back the way it was please do choose a > more unique name than "MAIN". Something prefixed with "GMK_" at least. OK, will do. > > Make never changes the I/O mode of its standard streams. And it > > shouldn't, since everything Make itself writes or reads is always > > plain text. Since the standard streams always start in text mode, > > your suggestion boils down to make input from the temporary file be > > always in text mode. > > Well, I assumed that something that invoked make could set the mode and > then make could inherit that mode. I don't know if that's how it works > or not in Windows. And of course I doubt anyone does that. The text/binary mode cannot be inherited the way file descriptors are. That mode is entirely private to each application, and the startup code provided by MS (to which MinGW is compatible) unconditionally initializes it to text mode. Each application decides on its own how it should treat its standard streams; most leave them at the default text mode, but some switch to binary. Examples of the latter are ports of 'cat' ("cat foo > bar" should create a file that is identical to 'foo'), carefully done ports of 'tr' and Sed (otherwise you cannot edit in or out CR characters), etc. > I understand your point. I just wonder if this difference might end up > being visible to the user in some way. If the users will see any difference, it means (barring bugs in Make) the child program used binary I/O where it would use text I/O had its stdout/stderr not been redirected. But this belongs to the broader class of problems with -O, not unlike the colored screen output that was discussed here a few days ago. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
64 posts
|
In reply to
this post by Paul Smith-20
On 05/05/2013 05:30 PM, Paul Smith wrote:
> On Sun, 2013-05-05 at 11:11 +0200, Stefano Lattarini wrote: >> Sorry to add this only now, but I realized the failure is only >> reproducible if I run the testsuite with "make -j", as in "make -j8 >> check; and even in that case, the failure is racy. With a bare "make >> check", things work for me as well. On the other hand, increasing the >> parallelism even more, other tests start to fail as well: > > The test suite definitely cannot be run in parallel. > I realize that (and it's no issue, considering how fast the make testsuite is); but I often run a "make -j8 check" just after running "./configure" (requires less typing than "make -j8 && make check"), and that's where things broke down. > However this > should not happen (and does not happen in my environment when I run the > commands above) because the test harness cleans out the environment, > which will remove any of the MAKEFLAGS or MFLAGS variables that might > tell the make to run in parallel when it's not expected. > > Can you examine your shell configuration files etc. to see if they're > setting MAKEFLAGS or MFLAGS? > Definitely not. > Although if that's true then the tests should fail all the time. > > Can you verify that there don't seem to be any leftover test files in > the tests directory? Sometimes if something doesn't get cleaned up > correctly that can cause future builds to fail. However if that were > the case then "make check" without -j would fail as well. > > I don't have an explanation for this. > It's no big deal; I can just run "make check" and live happily. Also, I don't have the mental bandwidth to try to debug this ATM, sorry. Let's return to the issue if I get bitten again. Or maybe you'll manage to reproduce the issue somehow. In the meantime, thanks for your help and your attention, Stefano _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
96 posts
|
On Sun, May 5, 2013 at 2:49 PM, Stefano Lattarini
< [hidden email]> wrote:
... [
show rest of quote]
Remember that GNU make has a .NOTPARALLEL special target to suppress parallelism. Maybe a makefile in the test suite would benefit from one of those? -David _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
|
1236 posts
|
In reply to
this post by Eli Zaretskii
... [
show rest of quote]
Done. _______________________________________________ Bug-make mailing list [hidden email] https://lists.gnu.org/mailman/listinfo/bug-make |
| Powered byNabble | Edit this page |
| Powered byNabble | Edit this page |
7万+
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
