Replacing ELF shared libraries

Update

2022-01-30

The linker flag --no-as-needed was previously omitted, but it is necessary with more recent toolchains (despite what the ld doc says about the default), at least in Debian and Fedora. I can’t find documentation for that, and I don’t remember whether the change in Fedora was distribution-specific.

There are plenty of words here about a rather simple trick(?), but since I haven’t come across it anywhere else, and have had it described as ‘clever’,In connexion with a failed attempt to provide some sanity in RHEL/Fedora linear algebra packaging. This is just a rehash.

it seems worth recording. TL;DR.‘Too long; didn’t read’, if you’re not down with the kids.

Suppose you have a well-defined library ABI, say for linear algebra (BLAS), and a canonical implementation of it, say ‘reference’ or ‘Netlib’ BLAS, which is linked into programs you run. Also, for some reason — probably performance — you want to replace that implementation without re-building those programs. You can do so by making use of the dynamic linker facilities. (This is with GNU Binutils, but I assume it will be similar on any ELF-based system.)

To be concrete, we have various computational programs linked against reference BLAS (libblas.so.3), which is an order of magnitude slower than, say, OpenBLASOr BLIS in the free software world. ATLAS seems to be dead, and you can definitely ignore it these days.

(libopenblas.so.0 from the Debian or EPEL packages), which we’d like programs to use transparently. In principle you could just export LD_PRELOAD=libopenblas.so.0, but that affects all programs, which at least isn’t clean.

So just symlink libopenblas.so.0 to libblas.so.3? That won’t work because libopenblas.so.0’s soname is wrong; it needs to be libblas.so.3 to be loaded in place of the original. It seems we need to rebuild OpenBLAS, redefining the library name.For which there is a hook in its build system.

Debian does that to satisfy its useful policy of interchangeable BLAS implementations.Actually, the rebuilt library is a subset of libopenblas, removing LAPACK and CBLAS.

Having done so, you can put the substitute in LD_LIBRARY_PATH globally or, better, do something like (on Debian)Debian actually uses its alternatives system to do the job, without affecting the loader’s search paths.

 # echo /usr/lib/x86\_64-linux-gnu/openblas-serial \
     >/etc/ld.so.conf.d/openblas.conf

Rebuilding like that is a pain, and perhaps we don’t need to, and can even be more general. Consider a dummy library that dlopens another one as required,Probably using a GCC constructor function.

possibly depending on the environment. You can do that, but the dummy one needs to define all the relevant symbols,FlexiBLAS does something like that, and is now enshrined in Fedora policy over my objections from experience using the technique described here in production on an HPC cluster.

and you might as well rebuild the library you want to load (in the simple case).

Instead, here’s the trick to build a trivial shim with the correct name to put on the load path to replace a dynamic library with an ABI-compatible alternative:

  cc -shared -o lib$n.so.$sover -Wl,-soname,lib$n.so.$sover -Wl,--no-as-needed -l$other

where $n is the name of the target library (e.g. blas), $sover is the dynamic library version (e.g. 3), and $other is the name of the substitute (e.g. openblas), assuming it doesn’t need extra libraries to link. You can check the result with readelf -d, looking for SONAME and NEEDED, which is all the meat.With no code, the shim is still  > 5000 kB stripped.

There’s a real example in the Fedora BLIS packaging (rather than OpenBLAS) which precedes the unfortunate Fedora policy. Of course, this works with a proprietary library that you can’t modify,Actually, modulo legalities, you could copy it and use
patchelf --set-soname,
but there’s some gotcha with that I don’t remember.

if you must.

The technique served well in production on an HPC cluster I used to manage. It was nearly all provisioned with RPM packaging, and most of the relevant packages linked reference BLAS. (I’m talking RHEL6.) All those got an immediate speed boost for linear algebra operations by subverting the BLAS dynamic libraries with libopenblas via ld.so.conf.There were more actually targets than libblas, primarily R’s libRblas, and also the equivalent for LAPACK, since OpenBLAS has various optimized LAPACK routines.

There were also OpenMP parallelized versions which you could use to boost R, in particular, by putting them on LD_LIBRARY_PATH. Knights Landing† systems were just appearing — Liverpool had one of the first in the UK — and there was no AVX512 support in OpenBLAS, but there was in BLIS after a while. OpenBLAS was generally preferable, but on the rather heterogeneous cluster, I could set up ld.so.conf to point to specific shims — openblas or blis — in the configuration for each node type.