Page MenuHomeSolus

Glibc 2.33 AVX2/Haswell hardware capabilities requires tooling changes.
Open, NormalPublic

Description

A few changes will have to be made to our tooling to switch to the new glibc HWCAPS capabilities.

AVX2/Haswell Hardware Capabilities

glibc have changed the install path from /usr/lib64/haswell to /usr/lib64/glibc-hwcaps/x86-64-v3/ (for our case) to be cross platform agnostic i.e. AMD inclusion, the old haswell path hasn't be removed yet but will be at some point (see news note below)

A future version of glibc will stop loading shared objects from the "tls" subdirectories on the library search path, the subdirectory that corresponds to the AT_PLATFORM system name, and also stop employing the legacy AT_HWCAP search mechanism. Applications should switch to the new glibc-hwcaps mechanism instead; if they do not do that, only the baseline version (directly from the search path directory) will be loaded.

gcc have introduced new arches definitions to match the glibc definitions e.g. -march=x86-64-v3 - https://gcc.gnu.org/onlinedocs/gcc/x86-Options.html

The closest match to x86-64-v3 is bdver4 (Excavator), this is actually a step down from haswell and we lose the following extensions: FSGSBASE PCLMUL RDRAND XSAVEOPT. This could potentially lead to a performance regression, but does include bdver4 processors. However, the AVX2 implementation of Excavator has a bad reputation from memory.

The relevant commit with everything nicely spelt out for us: https://gitlab.com/x86-psABIs/x86-64-ABI/-/commit/77566eb03bc6a326811cb7e9a6b9396884b67c7c

TODO:

  • Update gcc v11
  • Update LLVM stack to v12
  • YPKG2: when building with avx2 on change the %installlib% directory to /usr/lib64/glibc-hwcaps/x86-64-v3/
  • YPKG2: when building with avx2 on change the march to -march=x86-64-v3. Keep -mtune=haswell for now.
  • YPKG2: maybe a good idea to not run the check step on avx2 builds to avoid confusing segfaults for pc's without avx2 whilst we are here.
  • dracut: Also ignore /usr/lib64/glibc-hwcaps/x86-64-v3/ dir.
  • snapd: Allow libs from /usr/lib64/glibc-hwcaps/x86-64-v3/ to be loaded.
  • Rebuild every package that has avx2 libs enabled to the new paths:
    • glibc
    • openblas
    • gromacs
    • libflac
    • libwebp
    • pytorch
    • fftw3
    • graphene
    • openimagedenoise
    • openvdb
    • xxhash
    • ...more?
  • Verify these new paths are being loaded on Intel >=haswell and AMD >=zen (ldd).
  • Verify there isn't any performance regression with base glibc libs. /usr/lib64/glibc/benchmarks from glibc-devel (or use phoronix's pts/glibc-bench)
  • Verify there isn't any performance regression with the slightly different ARCH definitions for end applications (echo | gcc -dM -E - -march={haswell,x86_64-v3} to check).
  • Benchmark R via rbenchmark. It sees a decent uplift from openblas' avx2 libs to verify there isn't any performance regression.
  • Verify that the 32bit AVX2 stuff (/usr/lib32/glibc-hwcaps/x86-64-v3/) works as expected. Doesn't seem to be mentioned in the x86-64 psABI.

Event Timeline

joebonrichie changed the task status from Open to In Progress.Jan 3 2021, 10:42 AM
joebonrichie triaged this task as Normal priority.
joebonrichie created this task.
joebonrichie updated the task description. (Show Details)Jan 3 2021, 10:52 AM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 10:54 AM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 10:58 AM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 11:05 AM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 11:43 AM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 12:58 PM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 1:02 PM
joebonrichie updated the task description. (Show Details)Jan 3 2021, 1:34 PM
Jacalz awarded a token.Jan 3 2021, 1:38 PM
Jacalz added a subscriber: Jacalz.
joebonrichie renamed this task from Toolchain upgrade: GCC 10.3, binutils 2.35.1, glibc 2.32 to Glibc 2.33 AVX2/Haswell hardware capabilities requires tooling changes..Jan 3 2021, 1:55 PM
joebonrichie updated the task description. (Show Details)

Only just noticed these changes only take effect for glibc 2.33 (not released), we need to update to glibc 2.32 first. 🤦 This can wait a little longer, updated the doc the reflect the changes.

joebonrichie updated the task description. (Show Details)Jan 3 2021, 1:57 PM

Moved libxcrypt section to a different task. T9502.

joebonrichie changed the task status from In Progress to Open.Jan 3 2021, 2:09 PM
JoshStrobl updated the task description. (Show Details)Jan 3 2021, 6:34 PM
joebonrichie updated the task description. (Show Details)Mar 8 2021, 8:37 PM
joebonrichie updated the task description. (Show Details)Mar 11 2021, 8:43 PM
joebonrichie updated the task description. (Show Details)Apr 3 2021, 3:44 PM
joebonrichie updated the task description. (Show Details)Apr 8 2021, 11:18 AM
joebonrichie updated the task description. (Show Details)Aug 16 2021, 2:41 PM
DrSheppard added a subscriber: DrSheppard.
DataDrake moved this task from Backlog to Improvement on the Software board.Mar 12 2022, 11:19 PM