Discussion:
[gentoo-alt] clang-5 on macOS
Michael Weiser
2017-12-25 23:17:55 UTC
Permalink
Happy Christmas!

Is anyone (else) working to get clang-5 running in prefix on macOS?

After upgrading to macOS 10.13, a couple of packages stopped compiling
with clang 3.9.1. The errors came from system headers. It looked like
Apple had again used some funky new clang feature in them which 3.9.1
didn't support.

(Disclamer: I left prefix untouched when updating. It's still running
with CHOST=x86_64-apple-darwin16 and profile
prefix/darwin/macos/10.12/x64.)

Getting clang-5 up and running wasn't all that bad: I needed to clone
the libcxx and libcxxabi ebuilds for version 5.0.1, adjust eclass
cmake-utils and reintroduce some prefix provisions to clang and
compiler-rt ebuilds that didn't make the move from the llvm ebuild.

The cmake problem is somewhat tricky to explain: A special prefix
handling is supposed to force all cmake-based builds to link against the
libraries in prefix. This is done by setting an rpath and forcing cmake
to use that rpath whenever possible. Normally, cmake will link the
binaries for the build directory with an rpath that prefers libraries in
the build directory. This is disabled by setting
CMAKE_BUILD_WITH_INSTALL_RPATH.

For LLVM and clang builds this causes a problem: Newly compiled binaries
immediately get linked with the forced rpath. But the build calls some
of them while still in the build directory and before installation of
their libraries (especially llvm-tblgen and clang-tblgen). So they pick
up the old installed versions of the libraries via the rpath and croak
on missing symbols. I guess, other packages would face the same problem,
if they called dynamically linked binaries from the build directory -
which is somewhat rare.

Does anyone remember why CMAKE_BUILD_WITH_INSTALL_RPATH was necessary
for cmake ebuilds?

Disabling this setting led to another problem: LLVM and clang have all
their libs reference each other using @rpath. This is done relative to
the executable they're linked into which gets an rpath of
@executable_path/../lib for that purpose. This is fine as long as both
LLVM and clang are built in one sitting. With the new concept of first
building and installing LLVM and then building clang, the call to
clang-tblgen in the build directory again croaks. But this time it is
because dynamic LLVM libraries that get pulled in by the dynamic LLVM
libraries it's linked against can't be found because they're not on its
@rpath. So I had to force an *install* rpath of
${EPREFIX}/usr/lib/llvm/${SLOT}/lib.

With this I got llvm, clang and their other components installed. clang
defaulted to /usr/include for includes at first and compiler-rt compiled
against the Xcode SDK. With some research in the 3.9.1 ebuilds that was
quickly remedied.

The only real fallout came from the fact that my binutils-apple had LTO
and tapi support enabled. Both broke horribly which rendered ld64
defunct.

As long as neither USE tapi nor lto are enabled on binutils-apple, they
should be fine. Symlinks to size and nm will break and will need
adjustment to point to the new llvm install though.

I think I got LTO support sorted in an updated binutils-apple-8.2.1
ebuild. I have also taken care of LLVMs move into a slotted
$EPREFIX/usr/lib/llvm which should make them more robust to future
updates and moves.

But tapi is a real mess which shines a spotlight on a more fundamental
problem: Apple has yet to release the source of Xcode 9. So tapi-1.30
won't compile with clang-5.0.1 any more. A tapi-2.0.0 open source drop
on GitHub (https://github.com/ributzka/tapi) requires Apple-specific
additions to LLVM (ObjCMetadata). Those from the last released Apple
clang-8.0.0 are slightly incompatible to llvm-5.0.0 *and* this
tapi-2.0.0. I am currently digging my way through that.

In case anyone wonders: tapi is the library that supports SDK .tbd stub
files. So if ld64 has no tapi support it can't use recent Apple SDKs,
which for the purposes of a prefix install is no big deal, I guess. It
might even serve as a flag that some package is trying to use an SDK.

Any comments on my approach would be highly appreciated. Patches
attached for documentation.
--
Thanks,
Michael
Fabian Groffen
2017-12-26 10:36:52 UTC
Permalink
Post by Michael Weiser
Happy Christmas!
Is anyone (else) working to get clang-5 running in prefix on macOS?
After upgrading to macOS 10.13, a couple of packages stopped compiling
with clang 3.9.1. The errors came from system headers. It looked like
Apple had again used some funky new clang feature in them which 3.9.1
didn't support.
Yes, I noticed that too, see:
https://bugs.gentoo.org/632500
Post by Michael Weiser
(Disclamer: I left prefix untouched when updating. It's still running
with CHOST=x86_64-apple-darwin16 and profile
prefix/darwin/macos/10.12/x64.)
Getting clang-5 up and running wasn't all that bad: I needed to clone
the libcxx and libcxxabi ebuilds for version 5.0.1, adjust eclass
cmake-utils and reintroduce some prefix provisions to clang and
compiler-rt ebuilds that didn't make the move from the llvm ebuild.
Oh, I experienced failures to compile it on a 10.13 profile.
Post by Michael Weiser
The cmake problem is somewhat tricky to explain: A special prefix
handling is supposed to force all cmake-based builds to link against the
libraries in prefix. This is done by setting an rpath and forcing cmake
to use that rpath whenever possible. Normally, cmake will link the
binaries for the build directory with an rpath that prefers libraries in
the build directory. This is disabled by setting
CMAKE_BUILD_WITH_INSTALL_RPATH.
For LLVM and clang builds this causes a problem: Newly compiled binaries
immediately get linked with the forced rpath. But the build calls some
of them while still in the build directory and before installation of
their libraries (especially llvm-tblgen and clang-tblgen). So they pick
up the old installed versions of the libraries via the rpath and croak
on missing symbols. I guess, other packages would face the same problem,
if they called dynamically linked binaries from the build directory -
which is somewhat rare.
Does anyone remember why CMAKE_BUILD_WITH_INSTALL_RPATH was necessary
for cmake ebuilds?
I don't recall, seems like this indeed shouldn't be forced.
Post by Michael Weiser
Disabling this setting led to another problem: LLVM and clang have all
the executable they're linked into which gets an rpath of
@executable_path/../lib for that purpose. This is fine as long as both
LLVM and clang are built in one sitting. With the new concept of first
building and installing LLVM and then building clang, the call to
clang-tblgen in the build directory again croaks. But this time it is
because dynamic LLVM libraries that get pulled in by the dynamic LLVM
libraries it's linked against can't be found because they're not on its
@rpath. So I had to force an *install* rpath of
${EPREFIX}/usr/lib/llvm/${SLOT}/lib.
I assume this is install_name_tool-ing.
Post by Michael Weiser
With this I got llvm, clang and their other components installed. clang
defaulted to /usr/include for includes at first and compiler-rt compiled
against the Xcode SDK. With some research in the 3.9.1 ebuilds that was
quickly remedied.
The only real fallout came from the fact that my binutils-apple had LTO
and tapi support enabled. Both broke horribly which rendered ld64
defunct.
As long as neither USE tapi nor lto are enabled on binutils-apple, they
should be fine. Symlinks to size and nm will break and will need
adjustment to point to the new llvm install though.
Hmm...
Post by Michael Weiser
I think I got LTO support sorted in an updated binutils-apple-8.2.1
ebuild. I have also taken care of LLVMs move into a slotted
$EPREFIX/usr/lib/llvm which should make them more robust to future
updates and moves.
But tapi is a real mess which shines a spotlight on a more fundamental
problem: Apple has yet to release the source of Xcode 9. So tapi-1.30
won't compile with clang-5.0.1 any more. A tapi-2.0.0 open source drop
on GitHub (https://github.com/ributzka/tapi) requires Apple-specific
additions to LLVM (ObjCMetadata). Those from the last released Apple
clang-8.0.0 are slightly incompatible to llvm-5.0.0 *and* this
tapi-2.0.0. I am currently digging my way through that.
In case anyone wonders: tapi is the library that supports SDK .tbd stub
files. So if ld64 has no tapi support it can't use recent Apple SDKs,
which for the purposes of a prefix install is no big deal, I guess. It
might even serve as a flag that some package is trying to use an SDK.
Any comments on my approach would be highly appreciated. Patches
attached for documentation.
Seems like you got yourself pretty deep into it! clang-5 is the blocker
for supporting high sierra IMO. Can you now build 5 using 3.9.1?

I think the cmake patches are ok, I'll look into the ebuilds you
mention. I'll sync whatever I can find in this area.

Thanks,
Fabian
Post by Michael Weiser
--- portage/eclass/cmake-utils.eclass.orig 2017-12-25 22:56:39.589527684 +0100
+++ portage/eclass/cmake-utils.eclass 2017-12-25 23:09:19.207489713 +0100
@@ -588,9 +588,25 @@
SET(CMAKE_PREFIX_PATH "${EPREFIX}/usr" CACHE STRING "" FORCE)
SET(CMAKE_SKIP_BUILD_RPATH OFF CACHE BOOL "" FORCE)
SET(CMAKE_SKIP_RPATH OFF CACHE BOOL "" FORCE)
+ _EOF_
+
+ case ${PN} in
+ llvm|clang)
+ cat >> "${build_rules}" <<- _EOF_ || die
+ SET(CMAKE_INSTALL_RPATH "${EPREFIX}/usr/lib/llvm/${SLOT}/$(get_libdir)" CACHE STRING "" FORCE)
+ _EOF_
+ ;;
+
+ *)
+ cat >> "${build_rules}" <<- _EOF_ || die
SET(CMAKE_BUILD_WITH_INSTALL_RPATH TRUE CACHE BOOL "")
SET(CMAKE_INSTALL_RPATH "${EPREFIX}/usr/lib;${EPREFIX}/usr/${CHOST}/lib/gcc;${EPREFIX}/usr/${CHOST}/lib;${EPREFIX}/usr/$(get_libdir);${EPREFIX}/$(get_libdir)" CACHE STRING "" FORCE)
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE CACHE BOOL "" FORCE)
+ _EOF_
+ ;;
+ esac
+
+ cat >> "${build_rules}" <<- _EOF_ || die
SET(CMAKE_INSTALL_NAME_DIR "${EPREFIX}/usr/lib" CACHE STRING "" FORCE)
ENDIF (NOT APPLE)
--- portage/sys-devel/clang/clang-5.0.1.ebuild 2017-12-20 23:24:32.000000000 +0100
+++ portage/sys-devel/clang/clang-5.0.1.ebuild 2017-12-23 21:38:22.000000000 +0100
@@ -110,6 +110,13 @@
fi
}
+src_prepare() {
+ default
+
+ eapply "${FILESDIR}"/${PN}-5.0.1-darwin_prefix-include-paths.patch
+ eprefixify lib/Frontend/InitHeaderSearch.cpp
+}
+
multilib_src_configure() {
local llvm_version=$(llvm-config --version) || die
local clang_version=$(ver_cut 1-3 "${llvm_version}")
--- a/lib/Frontend/InitHeaderSearch.cpp
+++ b/lib/Frontend/InitHeaderSearch.cpp
@@ -233,6 +233,7 @@
break;
// FIXME: temporary hack: hard-coded paths.
AddPath("/usr/local/include", System, false);
break;
@@ -505,6 +506,7 @@
// Add the default framework include paths on Darwin.
if (HSOpts.UseStandardSystemIncludes) {
if (triple.isOSDarwin()) {
AddPath("/System/Library/Frameworks", System, true);
AddPath("/Library/Frameworks", System, true);
}
--- portage/sys-libs/compiler-rt/compiler-rt-5.0.1.ebuild 2017-12-20 21:40:08.000000000 +0100
+++ portage/sys-libs/compiler-rt/compiler-rt-5.0.1.ebuild 2017-12-23 21:56:33.000000000 +0100
@@ -77,6 +77,13 @@
-DCOMPILER_RT_BUILD_XRAY=OFF
)
+ if use prefix && [[ "${CHOST}" == *-darwin* ]] ; then
+ mycmakeargs+=(
+ # disable use of SDK for the system itself
+ -DDARWIN_macosx_CACHED_SYSROOT=/
+ )
+ fi
+
if use test; then
mycmakeargs+=(
-DLIT_COMMAND="${EPREFIX}/usr/bin/lit"
--
Fabian Groffen
Gentoo on a different level
Michael Weiser
2017-12-26 11:55:40 UTC
Permalink
Hi Fabian,
Post by Fabian Groffen
Post by Michael Weiser
After upgrading to macOS 10.13, a couple of packages stopped compiling
with clang 3.9.1. The errors came from system headers. It looked like
Apple had again used some funky new clang feature in them which 3.9.1
didn't support.
https://bugs.gentoo.org/632500
Ah, right, totally forgot about bison: I grabbed the workaround from
Homebrew some time back
(https://github.com/Homebrew/homebrew-core/issues/18591).
Post by Fabian Groffen
Post by Michael Weiser
Getting clang-5 up and running wasn't all that bad: I needed to clone
the libcxx and libcxxabi ebuilds for version 5.0.1, adjust eclass
cmake-utils and reintroduce some prefix provisions to clang and
compiler-rt ebuilds that didn't make the move from the llvm ebuild.
Oh, I experienced failures to compile it on a 10.13 profile.
Considering that the only real difference I can see between profiles is
the MACOS_DEPLOYMENT_TARGET, it might have been the rpath problem. It
failed to compile for me too until I removed
CMAKE_BUILD_WITH_INSTALL_RPATH from the cmake config. This indication is
that some binary (most likely llvm-tblgen) is called from the build
directory and croaks on the old LLVM libs it loads from EPREFIX/usr/lib
with missing symbols.
Post by Fabian Groffen
Post by Michael Weiser
Does anyone remember why CMAKE_BUILD_WITH_INSTALL_RPATH was necessary
for cmake ebuilds?
I don't recall, seems like this indeed shouldn't be forced.
I'll just go ahead and drop it from cmake-utils.eclass completely and
see what breaks.
Post by Fabian Groffen
Post by Michael Weiser
@rpath. So I had to force an *install* rpath of
${EPREFIX}/usr/lib/llvm/${SLOT}/lib.
I assume this is install_name_tool-ing.
Well, in a way: The libraries' ids get to be @rpath/<lib> as well. This
way, when linking into a binary, they don't bring the path to themselves
with them any more. The benefit is that an LLVM/clang installation can
be fully relocatable. The drawback is, that you need to muck around with
DYLD_{,FALLBACK_}LIBRARY_PATH, link every binary that links against
them with an rpath where they're on or compile a static rpath into them
as I did. The latter still doesn't propagate into a binary that links
against them. So you can dlopen() them and they'll find their
dependencies. But linking against them still requires putting an rpath
into the binary they're linked into. It's all somewhat sad.

I now realize that this might have been exactly the intention of the
cmake settings, especially CMAKE_INSTALL_NAME_DIR. But a.) we'd need to
adjust that for the llvm slot directory and b.) we'd need to disable it
for the build directory.

BTW: AFAIK all this isn't done using install_name_tool but directly when
linking the binaries. According to
https://cmake.org/cmake/help/v3.0/variable/CMAKE_BUILD_WITH_INSTALL_RPATH.html,
cmake normally links libs and executables twice: Once upon build with a
additions to rpath that make them consistently use the newly compiled
libraries in the build tree and then again upon install with the actual
install rpath. This would also change the library id (install name).

Here's what it looks like on my system right now:

# otool -D EPREFIX/usr/lib/llvm/5/lib/libLLVMSupport.dylib
EPREFIX/usr/lib/llvm/5/lib/libLLVMSupport.dylib:
@rpath/libLLVMSupport.dylib
# otool -L EPREFIX/usr/lib/llvm/5/lib/libLLVMSupport.dylib
EPREFIX/usr/lib/llvm/5/lib/libLLVMSupport.dylib:
@rpath/libLLVMSupport.dylib (compatibility version 0.0.0, current version 0.0.0)
EPREFIX/usr/lib/libncurses.6.dylib (compatibility version 6.0.0, current version 6.0.0)
EPREFIX/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
@rpath/libLLVMDemangle.dylib (compatibility version 0.0.0, current version 0.0.0)
EPREFIX/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1.0.0)
# otool -l EPREFIX/usr/lib/llvm/5/lib/libLLVMSupport.dylib
EPREFIX/usr/lib/llvm/5/lib/libLLVMSupport.dylib:
[...]
Load command 3
cmd LC_ID_DYLIB
cmdsize 56
name @rpath/libLLVMSupport.dylib (offset 24)
Load command 12
cmd LC_LOAD_DYLIB
cmdsize 56
name /usr/lib/libSystem.B.dylib (offset 24)
time stamp 2 Thu Jan 1 01:00:02 1970
current version 1252.0.0
compatibility version 1.0.0
Load command 13
cmd LC_LOAD_DYLIB
cmdsize 56
name @rpath/libLLVMDemangle.dylib (offset 24)
time stamp 2 Thu Jan 1 01:00:02 1970
current version 0.0.0
compatibility version 0.0.0
Load command 14
cmd LC_LOAD_DYLIB
cmdsize 72
name EPREFIX/usr/lib/libc++.1.dylib (offset 24)
[...]
Load command 16
cmd LC_DATA_IN_CODE
cmdsize 16
dataoff 1040248
datasize 440
Load command 17
cmd LC_RPATH
cmdsize 56
path EPREFIX/usr/lib/llvm/5/lib (offset 12)
Post by Fabian Groffen
Post by Michael Weiser
As long as neither USE tapi nor lto are enabled on binutils-apple, they
should be fine. Symlinks to size and nm will break and will need
adjustment to point to the new llvm install though.
Hmm...
I've put the current version of the updated ebuild into a new bug
#642292. Rationale for the changes are somewhat scattered in the bug
comments, ebuild, patch headers and comments in the patches. Let me know
if you'd like me to collect them in one place. :)

The main change is that I added a function and a shim binary to cctools
that goes looking for llvm tools and libs in the slot directories for
major versions 10 to 4 and EPREFIX/usr/bin directly. Since cctools
dlopen libLTO and exec() the tools, that should make them very robust
against updates.

ld64 is another story because it directly links against libLTO. If only
libLTO had a proper version number, portage could help by preserving it
on update. As of now, only the different slot paths of llvm major
versions might provide some protection against breakage.
Post by Fabian Groffen
Post by Michael Weiser
Any comments on my approach would be highly appreciated. Patches
attached for documentation.
Seems like you got yourself pretty deep into it! clang-5 is the blocker
for supporting high sierra IMO. Can you now build 5 using 3.9.1?
I have indeed successfully built llvm and clang 5.0.1 using 3.9.1 with
the above changes in my existing prefix. I am now trying to get all
packages updated to the current portage tree, switch profile and CHOST
to 10.13 and recompile the whole thing again. This will tell, if it's
self-hosting. After that I can try downgrading llvm/clang and upgrading
it again to see what breaks.
Post by Fabian Groffen
I think the cmake patches are ok, I'll look into the ebuilds you
mention. I'll sync whatever I can find in this area.
Actually, the cmake patch might be incorrect since it got lost on emaint
sync and I had to reconstruct it from memory. I'm sure about
CMAKE_BUILD_WITH_INSTALL_RPATH, CMAKE_INSTALL_RPATH,
CMAKE_INSTALL_RPATH_USE_LINK_PATH but I think I also dropped
CMAKE_INSTALL_NAME_DIR for llvm and clang.
--
bye, Michael
Michael Weiser
2017-12-26 20:35:40 UTC
Permalink
Hi,
Post by Michael Weiser
Post by Fabian Groffen
I think the cmake patches are ok, I'll look into the ebuilds you
mention. I'll sync whatever I can find in this area.
Actually, the cmake patch might be incorrect since it got lost on emaint
sync and I had to reconstruct it from memory. I'm sure about
CMAKE_BUILD_WITH_INSTALL_RPATH, CMAKE_INSTALL_RPATH,
CMAKE_INSTALL_RPATH_USE_LINK_PATH but I think I also dropped
CMAKE_INSTALL_NAME_DIR for llvm and clang.
After some more playing with cmake I think I got it: We do want a
minimal CMAKE_INSTALL_RPATH (with just the path this project's libraries
get installed to) and CMAKE_INSTALL_RPATH_USE_LINK_PATH because of
https://cmake.org/Wiki/CMake_RPATH_handling#Always_full_RPATH. This way
cmake will automatically add the directories of libraries that a project
links against and that are outside the build directory. No idea how the
first part can be done without help by the upstream source or the
individual ebuilds though.

We do most likely not want CMAKE_BUILD_WITH_INSTALL_RPATH and
CMAKE_INSTALL_NAME_DIR because we now know it causes breakage and the
benefit is still unclear.
--
bye, Micha
Michael Weiser
2017-12-26 23:17:30 UTC
Permalink
Me again,
Post by Michael Weiser
We do want a
minimal CMAKE_INSTALL_RPATH (with just the path this project's libraries
get installed to) and CMAKE_INSTALL_RPATH_USE_LINK_PATH because of
https://cmake.org/Wiki/CMake_RPATH_handling#Always_full_RPATH. This way
cmake will automatically add the directories of libraries that a project
links against and that are outside the build directory. No idea how the
first part can be done without help by the upstream source or the
individual ebuilds though.
From the binutils-config ldwrapper.c:

* On ELF platforms, the wrapper will then add -R (-rpath) entries for
* all -L entries found in the invocation to ensure the libraries found
* at link time will be found at runtime too.

That's basically what cmake does. We could do the same on Darwin. I
think the only reason it wasn't needed until now was that every single
dylib on Darwin normally brings its own path with it upon linking. This
would invalidate the need to fiddle with cmake-utils.eclass. It might
cause some other undesired behaviour though, like build directory paths
leaking into rpaths...

I will try the MACOSX_RPATH ON setting for llvm and clang. According to
above link this should disable the @rpath install names and get us back
the standard absolute paths in dylibs.
--
Bye,
Michael
Michael Weiser
2017-12-31 13:37:40 UTC
Permalink
Hi,
Post by Michael Weiser
* On ELF platforms, the wrapper will then add -R (-rpath) entries for
[...]
Post by Michael Weiser
I will try the MACOSX_RPATH ON setting for llvm and clang. According to
For completeness' sake and because bugs.gentoo.org doesn't show up on
search engines, here's the link to the sort-a tracker bug:
https://bugs.gentoo.org/642656.
--
Micha
Loading...