If you’re deploying C programs that rely on system
libraries, things may get tricky if you cross flavors or
versions of linux. But you can probably deploy static-linked
executables more easily than setting up docker.
For various reasons static linking isn't a good idea for C applications. glibc doesn't work well when statically linked. musl works well. But glibc has strong backward compatibility using symbol versioning. It's not usually worth the trouble to build and link aginst musl if your distro uses glibc. What you should be worried about are all the other third-party libraries, which aren't often written to be statically linked--because of namespace pollution, because of slightly different semantics between dynamic and static code, especially at application startup.
Fortunately, Linux supports packaging apps much like macOS bundles, where all shared objects are kept under a relocatable subtree. When building a binary that will be installed in bin/, for example, just use the following GCC flags:
The first flag tells the linker to find libraries in the lib/ directory adjacent to the binary file itself. The second flag tells the linker to try LD_LIBRARY_PATH first rather than the embedded rpath. Debugging and regression testing can be very difficult without the ability to use LD_LIBRARY_PATH, and unfortunately LD_LIBRARY_PATH has lower precedence than embedded rpaths, thus the need for --enable-new-dtags to change the default behavior.
Note that $ORIGIN is a special string that is expanded by the runtime linker, not by the shell at compile time. It's an unfortunate choice of syntax. Getting $ORIGIN to pass through to the compiler without being evaluated by a shell can be tricky when dealing with recursive make and shell invocations, such as from an RPM spec file.
Another example: when building a Lua module I'll use
-Wl,-rpath,'$ORIGIN/../..' -Wl,--enable-new-dtags
because Lua modules are usually installed under $(prefix)/lib/lua/5.?/, two directories below lib/.
And then you might just end up using setcap on it anyhow, perhaps to load privileged ports, which is treated like suid, making it so that the linker won't look at LD_LIBRARY_PATH, in which case you have to create a conf file with your paths and feed it to ldconfig after using ldd on the program because you're confused about why it suddenly doesn't load its libraries.
And there's another fun security restriction in there where chown erases any setcap permissions you had applied, just in case you had to adjust something and didn't understand why you suddenly lost your setcap permissions.
Why? If you control that, you likely also control PATH and others. There are lots of variables which mean game over if you have write access to them. And you also need a local fs write access to use it, so that's another step. I know it could be something to restrict, but it's really far down the list... (configuring selinux/apparmor would solve the same issue but in a more generic way)
Using -Wl,--enable-new-dtags merely restores the default behavior. More specifically, it only restores the usual precedence behavior.
Most people don't set the DT_RPATH ELF tag. It's usually considered best practice to NOT set DT_RPATH. And that's generally good advice, except that advice assumes absolute paths. Most people aren't familiar with $ORIGIN, which allows using relative rpaths. In any event, my point is that most environments already allow LD_LIBRARY_PATH. If it's been restricted then --enable-new-dtags won't change that.
What --enable-new-dtags actually does is set DT_RUNPATH instead of DT_RPATH. DT_RPATH has higher precedence than LD_LIBRARY_PATH; DT_RUNPATH has lower precedence.
To reiterate, DT_RPATH and DT_RUNPATH are independent of LD_LIBRARY_PATH, and don't effect whether it's permitted or not. I recommend --enable-new-dtags to avoid confusion and unintentional side-effects.
I've known about $ORIGIN for awhile, but only recently hit upon --enable-new-dtags. I was hacking on a project recently ported to and packaged for CentOS 7. The relevant components were a Lua C module (acme.foo) binding a shared library (libacme-foo). The in-tree unit and regression tests used LUA_PATH, LUA_CPATH, and LD_LIBRARY_PATH so the in-tree objects were loaded instead of any that might happen to be installed on the system. I was adding some debugging code to the shared library and running `make check`, but my changes didn't seem to be coming through in the tests. I was totally confused.
After far too much wasted time, I eventually realized that CentOS 7's base Lua 5.1 package was built with an embedded rpath of /usr/lib64. A version of the package I was hacking on was already installed on the system. So /usr/lib64/libacme-foo.so was linked when dlopen'ing the Lua C module instead of the in-tree libacme-foo.so that should have been found with LD_LIBRARY_PATH. That's how I became familiar with the precedence of rpath and LD_LIBRARY_PATH, and the availability of the --enable-new-dtags option.
I'm not sure why Red Hat built their core Lua 5.1 binary with an rpath, but they almost certainly should have used --enable-new-dtags when doing so. Notably, /usr/bin/perl on CentOS 7 is built with --enable-new-dtags; it has DT_RUNPATH=/usr/lib64/perl5/CORE. Just as notably, AFAICT Python isn't built with any rpath.
Fortunately, Linux supports packaging apps much like macOS bundles, where all shared objects are kept under a relocatable subtree. When building a binary that will be installed in bin/, for example, just use the following GCC flags:
The first flag tells the linker to find libraries in the lib/ directory adjacent to the binary file itself. The second flag tells the linker to try LD_LIBRARY_PATH first rather than the embedded rpath. Debugging and regression testing can be very difficult without the ability to use LD_LIBRARY_PATH, and unfortunately LD_LIBRARY_PATH has lower precedence than embedded rpaths, thus the need for --enable-new-dtags to change the default behavior.Note that $ORIGIN is a special string that is expanded by the runtime linker, not by the shell at compile time. It's an unfortunate choice of syntax. Getting $ORIGIN to pass through to the compiler without being evaluated by a shell can be tricky when dealing with recursive make and shell invocations, such as from an RPM spec file.
Another example: when building a Lua module I'll use
because Lua modules are usually installed under $(prefix)/lib/lua/5.?/, two directories below lib/.