[developer] Revised webrev for: 5196 The cw wrapper restricts gcc to -O2

Discussion:

Gary Mills via illumos-developer

2014-10-08 14:41:23 UTC

This is for illumos bug `5196 The cw wrapper restricts gcc to -O2'.

The webrev is here:

http://cr.illumos.org/~webrev/jgmills/5196-2

Unlike the previous version, this revision ensures that nothing
changes in illumos by retaining all of the optimization settings
used in the build. It does make it easier to change optimization
in the future.

The problem to be solved is that code in cw.c limits the optimization
level for gcc to 2 even though the default level in Makefile.master is
3. My approach was to remove that limit from cw.c and to transfer
any necessary gcc optimization settings and adjustments to Makefiles.

In this revision, I've modified the cw wrapper so that it does not
limit gcc optimization to level 2 and so that it does not set any gcc
optimization adjustments through `-f' options. The wrapper now only
maps `-xOn' for cc to `-On' for gcc, although this mapping is no
longer necessary.

I've also added separate optimization settings for cc and gcc to
Makefiles, principally to Makefile.master . The defaults are the same
as were coded in the file or produced by the cw wrapper. As well,
I've identified all Makefiles that change optimization away from the
defaults, and modified them similarly.

To ensure that none of the binaries had changed in this revision, I
compared prototype areas from before and after the changes with
`wsdiff'. In some cases, I also verified that the cw and gcc command
lines were unchanged.

I was using Illumos gcc-4.4.4-il-3 as the primary compiler, with the
Sun Studio compiler as shadow.

The original bug report is here:

https://illumos.org/issues/5196
--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Igor Kozhukhov via illumos-developer

2014-10-08 14:54:54 UTC

Permalink

I¹m not sure we need specify ADJUST_OPT1 or ADJUST_OPT2 or something else.
Will be better provide the same flags what provided by cw wrapper now and
update it later if needed.
Because for different GCC versions or another compilers we need provide
different list of flags to uts or cmd or lib targets.

--
Best regards,
Igor Kozhukhov

On 08/10/14 18:41, "Gary Mills via illumos-developer"

Post by Gary Mills via illumos-developer
This is for illumos bug `5196 The cw wrapper restricts gcc to -O2'.
http://cr.illumos.org/~webrev/jgmills/5196-2
Unlike the previous version, this revision ensures that nothing
changes in illumos by retaining all of the optimization settings
used in the build. It does make it easier to change optimization
in the future.
The problem to be solved is that code in cw.c limits the optimization
level for gcc to 2 even though the default level in Makefile.master is
3. My approach was to remove that limit from cw.c and to transfer
any necessary gcc optimization settings and adjustments to Makefiles.
In this revision, I've modified the cw wrapper so that it does not
limit gcc optimization to level 2 and so that it does not set any gcc
optimization adjustments through `-f' options. The wrapper now only
maps `-xOn' for cc to `-On' for gcc, although this mapping is no
longer necessary.
I've also added separate optimization settings for cc and gcc to
Makefiles, principally to Makefile.master . The defaults are the same
as were coded in the file or produced by the cw wrapper. As well,
I've identified all Makefiles that change optimization away from the
defaults, and modified them similarly.
To ensure that none of the binaries had changed in this revision, I
compared prototype areas from before and after the changes with
`wsdiff'. In some cases, I also verified that the cw and gcc command
lines were unchanged.
I was using Illumos gcc-4.4.4-il-3 as the primary compiler, with the
Sun Studio compiler as shadow.
https://illumos.org/issues/5196
--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-
-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
https://www.listbox.com/member/archive/rss/182179/21175093-0a5a4566
https://www.listbox.com/member/?&
4849
Powered by Listbox: http://www.listbox.com

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Gary Mills via illumos-developer

2014-10-08 15:51:02 UTC

Permalink

Post by Igor Kozhukhov via illumos-developer
I¹m not sure we need specify ADJUST_OPT1 or ADJUST_OPT2 or something else.
Will be better provide the same flags what provided by cw wrapper now and
update it later if needed.

Those _are_ exactly the same flags that were provided by the cw
wrapper.

Post by Igor Kozhukhov via illumos-developer
Because for different GCC versions or another compilers we need provide
different list of flags to uts or cmd or lib targets.

--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Keith Wesolowski via illumos-developer

2014-10-08 15:57:01 UTC

Permalink

Post by Igor Kozhukhov via illumos-developer
Because for different GCC versions or another compilers we need provide
different list of flags to uts or cmd or lib targets.

Only one version of gcc is supported. Changing the gate to support a
different version requires changing many of these macros anyway.

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Keith Wesolowski via illumos-developer

2014-10-08 16:14:57 UTC

Permalink

I don't believe that this is correct, although whether that's the case
depends on what you think cw's purpose is.

If cw is a general-purpose compiler argument translator, then maybe this
is reasonable, although I still think that what I pointed out previously
applies here: you need to translate the semantics of these levels, not
just pass through an arbitrary number. But this means that everyone
writing makefiles has to be very careful not to clobber COPTFLAG. The
fact that you had to change so many makefiles precisely for this reason
suggests to me that this aspect of the change is somewhat hazardous.
Maybe cw should respect an environment variable CW_NOT_ON=1 that allows
it to emit optimisation levels beyond 2 and/or without the extra flags,
if this is considered a valid use case.

If cw is just a tool for building ON, which would be consistent with the
fact that ON does not deliver it nor does any distributor I'm aware of,
then this is not correct. In this view, the purpose of cw is to
translate Studio options that may appear in ON makefiles (not
necessarily any others) into a set of gcc options (not necessarily
semantically equivalent) that are known to successfully build a correct
ON. If this is how we think of cw, then it is a bug for cw to ever emit
'-O3' because it is known to break ON.

What this actually looks like to me, which is exciting and long overdue
but very very incomplete, is the eventual conversion of gcc in makefiles
to a first-class citizen. Presumably at that point cw is either
eliminated along with Studio support or kept on as an unsupported
transition aid for whoever is still using Studio. I'd be totally
thrilled to see this, but you have to do the whole thing. I think it's
less work than you realise.

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Gary Mills via illumos-developer

2014-10-08 18:02:22 UTC

Permalink

Post by Keith Wesolowski via illumos-developer

[...]

Post by Keith Wesolowski via illumos-developer

Post by Gary Mills via illumos-developer
In this revision, I've modified the cw wrapper so that it does not
limit gcc optimization to level 2 and so that it does not set any gcc
optimization adjustments through `-f' options. The wrapper now only
maps `-xOn' for cc to `-On' for gcc, although this mapping is no
longer necessary.

I don't believe that this is correct, although whether that's the case
depends on what you think cw's purpose is.

What I meant and should have said is that the mapping is no longer
needed because Makefiles specify cc and gcc optimization explicitly,
and that it only remains as a fallback.

Post by Keith Wesolowski via illumos-developer
If cw is a general-purpose compiler argument translator, then maybe this
is reasonable, although I still think that what I pointed out previously
applies here: you need to translate the semantics of these levels, not
just pass through an arbitrary number.

As others have said, this is impossible. There is no exact
correspondence between cc and gcc optimization levels, even if details
are taken into account. What you suggest here can't be done even
in approximation.

Post by Keith Wesolowski via illumos-developer
But this means that everyone
writing makefiles has to be very careful not to clobber COPTFLAG. The
fact that you had to change so many makefiles precisely for this reason
suggests to me that this aspect of the change is somewhat hazardous.

You always have to be careful in replacing variables defined in
Makefile.master . For example, I found 18 Makefiles that redefine
CFLAGS, even though you are not supposed to do that. At least the lib
directory has README.Makefiles to give some guidance. I don't know of
any other similar documents.

Post by Keith Wesolowski via illumos-developer
Maybe cw should respect an environment variable CW_NOT_ON=1 that allows
it to emit optimisation levels beyond 2 and/or without the extra flags,
if this is considered a valid use case.

I don't know that that would help. It might worsen the situation,
instead.

Post by Keith Wesolowski via illumos-developer
If cw is just a tool for building ON, which would be consistent with the
fact that ON does not deliver it nor does any distributor I'm aware of,
then this is not correct. In this view, the purpose of cw is to
translate Studio options that may appear in ON makefiles (not
necessarily any others) into a set of gcc options (not necessarily
semantically equivalent) that are known to successfully build a correct
ON. If this is how we think of cw, then it is a bug for cw to ever emit
'-O3' because it is known to break ON.

I'd say that it is a tool for building ON, but it's also a tool that
can't be fixed by just sharpening it. As well, some parts of ON work
fine with `gcc -O3'. My changes facilitate this work.

Post by Keith Wesolowski via illumos-developer
What this actually looks like to me, which is exciting and long overdue
but very very incomplete, is the eventual conversion of gcc in makefiles
to a first-class citizen. Presumably at that point cw is either
eliminated along with Studio support or kept on as an unsupported
transition aid for whoever is still using Studio. I'd be totally
thrilled to see this, but you have to do the whole thing. I think it's
less work than you realise.

Perhaps what I'm proposing now is a step in that direction, but a step
that that leaves the build intact.
--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Gary Mills via illumos-developer

2014-10-08 18:26:57 UTC

Permalink

It strikes me that with all of my Makefile changes in this revision, I
could leave cw.c unchanged. After all, the `-xOn' option is no longer
present in any of the Makefiles. We have two choices. Which is the
best now? With my change to cw.c, if some Makefile happens to specify
`cw -xO3', cw will map it to `gcc -O3'. This may or may not work, but
will typically produce gcc warnings. With the original cw.c, it will
map it to `gcc -O2' along with some `-f' options. This likely will
work, but will likely also give a lower optimization level than the
writer intended.
--
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-

-------------------------------------------
illumos-developer
Archives: https://www.listbox.com/member/archive/182179/=now
RSS Feed: https://www.listbox.com/member/archive/rss/182179/21175072-86d49504
Modify Your Subscription: https://www.listbox.com/member/?member_id=21175072&id_secret=21175072-abdf7b7e
Powered by Listbox: http://www.listbox.com

Gary Mills via illumos-developer

2014-10-10 15:40:23 UTC

Permalink

Who can I list as a reviewer for these changes? So far, I've only
received a few questions. I've responded to them, sometimes with more
questions, but never got a response.

Post by Gary Mills via illumos-developer
The problem to be solved is that code in cw.c limits the optimization
level for gcc to 2 even though the default level in Makefile.master is
3. My approach was to remove that limit from cw.c and to transfer
any necessary gcc optimization settings and adjustments to Makefiles.
In this revision, I've modified the cw wrapper so that it does not
limit gcc optimization to level 2 and so that it does not set any gcc
optimization adjustments through `-f' options. The wrapper now only
maps `-xOn' for cc to `-On' for gcc, although this mapping is no
longer necessary.
I've also added separate optimization settings for cc and gcc to
Makefiles, principally to Makefile.master . The defaults are the same
as were coded in the file or produced by the cw wrapper. As well,
I've identified all Makefiles that change optimization away from the
defaults, and modified them similarly.
To ensure that none of the binaries had changed in this revision, I
compared prototype areas from before and after the changes with
`wsdiff'. In some cases, I also verified that the cw and gcc command
lines were unchanged.
I was using Illumos gcc-4.4.4-il-3 as the primary compiler, with the
Sun Studio compiler as shadow.
https://illumos.org/issues/5196

Gary Mills via illumos-developer

2014-10-21 14:08:59 UTC

Permalink

So, should I abandon this project? I thought that doing it this
way was an improvement to the build process and one that had no
negative effects. So far, I've had comments on the revision, but
nobody said they'd support it.