Page MenuHomeSolus

Cannot use Clang to build C++ extensions for python on Solus
Open, WishlistPublic

Description

I tried to use clang as the compiler to build a C++ extension library for python. It failed with the following error message:

ld.lld: error: unknown argument: --copy-dt-needed-entries
ld.lld: error: unknown argument: --copy-dt-needed-entries
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang++' failed with exit status 1

This behavior can be easily reproduced with the following setup:

Create a cpp file named demo.cpp with the following content:

// demo.cpp
#include <stdio.h>

int main()
{
    return 0;
}

In the same directory, create setup.py with the following content:

from distutils.core import setup, Extension

module1 = Extension('demo',
                    sources = ['demo.cpp'])

setup (name = 'PackageName',
       version = '1.0',
       description = 'This is a demo package',
       ext_modules = [module1])

Then build the extension with the following command:

CC=clang CXX=clang++ python3 setup.py build

The output on my laptop is as follows:

running build
running build_ext
building 'demo' extension
creating build
creating build/temp.linux-x86_64-3.6
clang -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -mtune=generic -march=x86-64 -g2 -O2 -pipe -fPIC -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-strong --param ssp-buffer-size=32 -fasynchronous-unwind-tables -ftree-vectorize -feliminate-unused-debug-types -Wall -Wno-error -Wp,-D_REENTRANT -I/usr/include -mtune=generic -march=x86-64 -g2 -O2 -pipe -fPIC -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-strong --param ssp-buffer-size=32 -fasynchronous-unwind-tables -ftree-vectorize -feliminate-unused-debug-types -Wall -Wno-error -Wp,-D_REENTRANT -fPIC -I/usr/include/python3.6m -c demo.cpp -o build/temp.linux-x86_64-3.6/demo.o
creating build/lib.linux-x86_64-3.6
clang++ -pthread -shared -Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions -Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions build/temp.linux-x86_64-3.6/demo.o -L/usr/lib64 -lpython3.6m -o build/lib.linux-x86_64-3.6/demo.cpython-36m-x86_64-linux-gnu.so
ld.lld: error: unknown argument: --copy-dt-needed-entries
ld.lld: error: unknown argument: --copy-dt-needed-entries
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'clang++' failed with exit status 1

I spent some time understanding what could cause the issue. It seems that this is because the flag --copy-dt-needed-entries was used when the python and python3 packages were compiled on Solus. When setuptools is used to compile C and C++ extensions, this flag is carried over. I checked the flags used to compile python3 with the following command:

$ python3 -m sysconfig | grep LDFLAGS
 	CONFIGURE_LDFLAGS = "-Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions"
	CONFIGURE_LDFLAGS_NODIST = ""
	CONFIG_ARGS = "'--prefix=/usr' '--build=x86_64-solus-linux' '--libdir=/usr/lib64' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var' '--libexecdir=/usr/lib64/python3' '--enable-shared' '--with-system-expat' '--with-system-ffi' '--with-threads' '--with-ensurepip=install' '--without-cxx-main' '--with-signal-module' '--enable-ipv6=yes' '--enable-loadable-sqlite-extensions' 'CFLAGS=-mtune=generic -march=x86-64 -g2 -O2 -pipe -fPIC -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -fstack-protector-strong --param ssp-buffer-size=32 -fasynchronous-unwind-tables -ftree-vectorize -feliminate-unused-debug-types -Wall -Wno-error -Wp,-D_REENTRANT -I/usr/include' 'build_alias=x86_64-solus-linux' 'CC=x86_64-solus-linux-gcc' 'LDFLAGS=-Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions'"
	LDFLAGS = "-Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions -Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions"
	PY_CORE_LDFLAGS = "-Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions -Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions"
	PY_LDFLAGS = "-Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions -Wl,--copy-dt-needed-entries -Wl,-O1 -Wl,-z,relro -Wl,-z,now -Wl,-z,max-page-size=0x1000 -Wl,-Bsymbolic-functions"
	PY_LDFLAGS_NODIST = ""

It can be seen that --copy-dt-needed-entries is included in the LDFLAGS.

My thought is that if we only use the flags that can be understood by both GNU compiler and llvm compiler when building python and python3, this problem will go away.

xulongwu4 created this task.Feb 2 2019, 6:02 PM
xulongwu4 updated the task description. (Show Details)Feb 2 2019, 6:06 PM
xulongwu4 edited projects, added Software; removed Lacks Project.
xulongwu4 updated the task description. (Show Details)
xulongwu4 updated the task description. (Show Details)Feb 2 2019, 6:11 PM
xulongwu4 updated the task description. (Show Details)Feb 2 2019, 6:14 PM
xulongwu4 updated the task description. (Show Details)
xulongwu4 updated the task description. (Show Details)

you did set clang: yes inside the package.yml?

xulongwu4 added a comment.EditedFeb 16 2019, 6:51 PM

you did set clang: yes inside the package.yml?

I did this. Besides, this is only relevant when submitting a patch for solus with solbuild. I also tried building the extension for myself on my laptop using the system libraries. I did install clang and other dependencies as well.

Whether I did this in solbuild or on my native system, I got the same error message. Thus I know that this flag is not set by solbuild. Rather it is set by the options that we used to build python, as I detailed in the earlier post.

To clarify: this isn't a problem with the compiler. It's python's fault for using the LLVM LLD linker instead of the standard GNU LD linker.

DataDrake triaged this task as Wishlist priority.Tue, Apr 30, 2:21 PM
DataDrake moved this task from Backlog to Package Fixes on the Software board.