Use dont_dlopen=true with GR libraries #5784

mkitti · 2022-11-01T23:23:15Z

GR uses a Preferences.jl mechanism to dlopen the libraries so the GR_jll does not need to dlopen the binaries directly:
https://github.com/jheinen/GR.jl/blob/f224e831abbade86ad1932a8b4b7fa2c85cec86d/src/funcptrs.jl#L33-L35

On Linux in certain configurations, a LD_PRELOAD is needed to get around this since the GR_jll binaries may be loaded before the system binaries. This pull request resolves this, but having GR_jll not load the binaries.

I'm also looking into using Overrides.toml or local preferences for overrides. Even with JLL overrides, this is still needed in case someone wants to use a Plots.jl backend that is not GR. In that case, using dlopen on GR_jll would just add time to loading. Additionally, GR_jll may interfere with other binaries making it difficult to load another backend like PyPlot.jl.

xref: jheinen/GR.jl#483 (comment)
xref: sciapp/gr#141 (comment)

Do we need a version bump for this?

We're probably going to need a version bump for this. Alternatively, I'm looking into using Overrides.toml

giordano · 2022-11-01T23:27:44Z

CC @jheinen

jheinen · 2022-11-02T02:51:32Z

As long as this patch does not affect the precompile behavior of Plots, feel free to merge it. To be honest, I have no idea how to test it before it is merged.

Explicitly setting the LD_PRELOAD path is a poor workaround and can't be the "final solution".

mkitti · 2022-11-02T03:25:44Z

To be clear this was originally proposed by @sjkelly and currently exists on the master branch of GR.jl:
https://github.com/jheinen/GR.jl/blob/master/build_tarballs.jl#L81-L84

I'm just running with it here.

Originally, the only way I knew how to defer the loading of the binaries was to defer the loading of GR_jll, but that broke precompilation.

This allows us to load GR_jll normally while still deferring the actual loading of the binaries.

The deferment is important because that gives us the chance to load alternative binaries or not load any binaries at all. The chance allows to build a robust configuration stage, to choose binaries, or to even repair the config.

vchuravy · 2022-11-02T11:17:14Z

An alternative is to use the existing methods in jll to swap out libraries.

I started to add functionality like set_library in https://github.com/cesmix-mit/LAMMPS.jl/blob/main/src/LAMMPS.jl#L30 to my packages.

sjkelly · 2022-11-02T12:44:17Z

I think the idea here is that Plots.jl ships GR by default, but one may not necessarily want to use use GR as the default backend in e.g. a sysimg. Making the loading deferred may have some startup and distribution size improvements.

…

On Wed, Nov 2, 2022, 07:17 Valentin Churavy ***@***.***> wrote: An alternative is to use the existing methods in jll to swap out libraries. I started to add functionality like set_library in https://github.com/cesmix-mit/LAMMPS.jl/blob/main/src/LAMMPS.jl#L30 to my packages. — Reply to this email directly, view it on GitHub <#5784 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAP5MDCVEPVYOSKDWQ73YF3WGJEUJANCNFSM6AAAAAARUQ7QEA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

t-bltg · 2022-11-02T14:18:21Z

To be honest, I have no idea how to test it before it is merged.

$ julia -e 'using Pkg; pkg"add BinaryBuilder JLLWrappers"'
$ julia build_tarballs.jl x86_64-linux-gnu --verbose --deploy=local
[...]
$ julia -e 'using JLLWrappers; JLLWrappers.dev_jll("GR")'
[...]
[ Info: GR_jll dev\'ed out to ~/.julia/dev/GR_jll with pre-populated override directory
$ julia -e 'using Pkg; Pkg.develop(path="~/.julia/dev/GR_jll")'
[...]
$ julia -e 'using Plots; png(plot(1:2), "foo")'  # looks good (uses libGR.so)
$ julia -e 'using Plots; x = y = -3:0.1:3; png(surface(x, y, (x, y) -> sinc(sqrt(x^2 + y^2))), "bar")'  # looks good (uses libGR3.so)
$ julia -e 'using Plots; display(plot(1:2)); println("press any key"); readline()'  # OK, uses libGKS.so

mkitti · 2022-11-02T14:48:21Z

An alternative is to use the existing methods in jll to swap out libraries.

Yes, I'm also pursuing the Preferences.jl based overrides system. I see this as complementary to that approach.

The overrides system let's us switch binaries, but what if we do not need to load either binary at all because we are using another Plots backend?

jheinen/GR.jl#483

t-bltg · 2022-11-02T14:55:00Z

The overrides system let's us switch binaries, but what if we do not need to load either binary at all because we are using another Plots backend?

Currently, GR is a hard Plots dependency and always imported.

mkitti · 2022-11-02T15:38:17Z

Currently, GR is a hard Plots dependency and always imported.

Exactly, but when are the binaries actually loaded via dlopen? Currently, this is done via GR_jll.jl which is also currently always loaded. However, after this pull request is merged, GR_jll.jl will not dlopen the binaries, leaving it to GR.jl.

If GR_jll.jl does not dlopen the binaries, when will GR.jl load the binaries? In GR.jl this is done via load_libs(). load_libs() is called from get_func_ptr(...). get_func_ptr(...) is called from libGR_ptr(...). libGR_ptr(...) is only called when a plotting function is called such as initgr(). Note that GR.jl currently has no __init__() function itself, although some of the submodules do.

To summarize, the call chain is as follows.

Low-level plotting function (e.g. GR.polyline)
GR.libGR_ptr
GR.get_func_ptr
GR.load_libs
Libdl.dlopen

In summary, when Plots.jl does import GR, GR.jl itself does not dlopen any of the binaries until someone actually tries to plot. Currently, GR_jll.jl does dlopen the binaries on load, but this pull request will address that. After this pull request, nothing will dlopen the binaries until the user tries to plot. If the user switches backends, the GR binaries will never be loaded via dlopen!

sjkelly · 2022-11-02T16:36:35Z

Slightly tangential... @mkitti this is a really nice summary of your design and dlsym caching mechanism. I did some benchmarks a while ago when I was looking at it, and it was something like 10x faster than normal dlsym. It may be good to generalize it for other packages that want to use these techniques.

mkitti · 2022-11-02T16:37:57Z

As long as this patch does not affect the precompile behavior of Plots, feel free to merge it. To be honest, I have no idea how to test it before it is merged.

Explicitly setting the LD_PRELOAD path is a poor workaround and can't be the "final solution".

@jheinen , to clarify, this pull request resolves the LD_PRELOAD issue when trying to configure GR.jl with an environment variable. Can we get your approval to move forward with this?

mkitti · 2022-11-02T16:48:02Z

Slightly tangential... @mkitti this is a really nice summary of your design and dlsym caching mechanism. I did some benchmarks a while ago when I was looking at it, and it was something like 10x faster than normal dlsym. It may be good to generalize it for other packages that want to use these techniques.

Thanks! The one reservation I have about this exact approach is that is very lazy and uses a mutable struct as a function pointer cache. An alternative approach would be load all the symbols at once into an immutable struct. I believe that may save some indirection later during normal use.

For GR.jl and Plots.jl, I think maximum laziness is the right approach. I'm not sure if other packages would want to be as lazy.

For example in JavaCall.jl, all the symbols are loaded into an immutable struct. https://github.com/JuliaInterop/JavaCall.jl/blob/63026e4cc6530c65b791519f34812e84ec843b14/src/jnienv.jl#L3
The loading occurs here:
https://github.com/JuliaInterop/JavaCall.jl/blob/575f4514a1604c8c880667ee6e1398ecdf94647c/src/JNI.jl#L137

What I would want to automate in the future is extraction of the symbol table for the shared library.

jheinen · 2022-11-02T18:04:34Z

@jheinen , to clarify, this pull request resolves the LD_PRELOAD issue when trying to configure GR.jl with an environment variable. Can we get your approval to move forward with this?

@mkitti : I agree with the PR, but I have no rights to merge it.

mkitti · 2022-11-02T18:12:08Z

@jheinen , to clarify, this pull request resolves the LD_PRELOAD issue when trying to configure GR.jl with an environment variable. Can we get your approval to move forward with this?

@mkitti : I agree with the PR, but I have no rights to merge it.

Thanks for checking in.

I'm currently testing via:
julia --project=. build_tarballs.jl --deploy=local x86_64-linux-gnu

I'm going to finish some local testing, and then I will ping Mose.

mkitti · 2022-11-02T20:18:09Z

Loading GR_jll locally works for basic plotting.

(tmp.mFM5iDWhYr) pkg> add GR
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  No Changes to `/tmp/tmp.mFM5iDWhYr/Project.toml`
  No Changes to `/tmp/tmp.mFM5iDWhYr/Manifest.toml`

(tmp.mFM5iDWhYr) pkg> dev GR_jll
   Resolving package versions...
    Updating `/tmp/tmp.mFM5iDWhYr/Project.toml`
  [d2c73de3] ~ GR_jll v0.69.1+0 ⇒ v0.69.1+1 `~/.julia/dev/GR_jll`
    Updating `/tmp/tmp.mFM5iDWhYr/Manifest.toml`
  [d2c73de3] ~ GR_jll v0.69.1+0 ⇒ v0.69.1+1 `~/.julia/dev/GR_jll`

julia> using GR
[ Info: Precompiling GR [28b8d3ca-fb5f-59d9-8090-bfdbd6d07a71]

julia> x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]; y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
6-element Vector{Float64}:
 0.3
 0.5
 0.4
 0.2
 0.6
 0.7

julia> plot(x,y)

giordano · 2022-11-02T20:20:00Z

But are you actually calling into libgr provided by gr_jll? Where does the manual dlopen happen? Edit: nevermind, you shared the link above: https://github.com/jheinen/GR.jl/blob/5cf55bcf5e99653ff79e827ebfc8c073515744fb/src/funcptrs.jl#L33-L35

giordano · 2022-11-02T20:22:41Z

Setting PATH globally looks like a bad idea: https://github.com/jheinen/GR.jl/blob/5cf55bcf5e99653ff79e827ebfc8c073515744fb/src/funcptrs.jl#L38

t-bltg · 2022-11-02T20:25:36Z

I agree, we tried multiple times to remove that before, but without success: removing it breaks Plots on windoze.

mkitti · 2022-11-02T20:44:08Z

Looks like I can change libraries after loading the module if I set the GRDIR environment variable and call GR.GRPreferences.__init__(). I'm developing both GR and GR_jll here.

julia> rm("gr/lib", force = true, recursive = true)

julia> using GR

julia> GR.GRPreferences.diagnostics()
┌ Info: GRDIR Environment Variable
└   get(ENV, "GRDIR", missing) = missing
┌ Info: GR Preferences
│   binary = "system"
└   grdir = "/tmp/tmp.mFM5iDWhYr/gr"
┌ Info: resolved_grdir
│   isdir(resolved_grdir) = true
└   isdir.(joinpath.((resolved_grdir,), ("bin", "lib", "include", "fonts"))) = (true, false, true, true)
┌ Info: GR_jll Preferences
│   libGR_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR"
│   libGR3_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR3"
│   libGRM_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGRM"
│   libGKS_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGKS"
└   gksqt_path = "/tmp/tmp.mFM5iDWhYr/gr/bin/gksqt"
┌ Info: GR_jll Override.toml
│   override_toml_path = "/home/mkitti/.julia/artifacts/Overrides.toml"
│   isfile(override_toml_path) = true
│   gr_jll_override_dict =
│    Dict{String, Any} with 1 entry:
└      "GR" => "/tmp/tmp.mFM5iDWhYr/gr"
┌ Info: GR_jll
│   GR_jll.libGR_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR"
│   GR_jll.libGR3_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR3"
│   GR_jll.libGRM_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGRM"
│   GR_jll.libGKS_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGKS"
└   GR_jll.gksqt_path = "/tmp/tmp.mFM5iDWhYr/gr/bin/gksqt"
(binary = "system", grdir = "/tmp/tmp.mFM5iDWhYr/gr", libGR_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR", libGR3_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR3", libGRM_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGRM", libGKS_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGKS", gksqt_path = "/tmp/tmp.mFM5iDWhYr/gr/bin/gksqt", override_toml_path = "/home/mkitti/.julia/artifacts/Overrides.toml", gr_jll_override_dict = Dict{String, Any}("GR" => "/tmp/tmp.mFM5iDWhYr/gr"))

julia> GR.GRPreferences.use_upstream_binary(joinpath(pwd(), "newgr"); override = (:project, :depot), force = true)
┌ Info: Downloading pre-compiled GR 0.69.1 Ubuntu binary
└   file = "/tmp/jl_J47gtF/gr-0.69.1-Ubuntu-x86_64.tar.gz"
┌ Info: grdir
└   grdir = "/tmp/tmp.mFM5iDWhYr/newgr/gr"
[ Info: Please restart Julia to change the GR binary configuration.

julia> ENV["GRDIR"] = "/tmp/tmp.mFM5iDWhYr/newgr/gr"
"/tmp/tmp.mFM5iDWhYr/newgr/gr"

julia> x = [0, 0.2, 0.4, 0.6, 0.8, 1.0]; y = [0.3, 0.5, 0.4, 0.2, 0.6, 0.7]
6-element Vector{Float64}:
 0.3
 0.5
 0.4
 0.2
 0.6
 0.7

julia> plot(x,y)
ERROR: could not load library "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR"
/tmp/tmp.mFM5iDWhYr/gr/lib/libGR.so: cannot open shared object file: No such file or directory
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl ./libdl.jl:117
  [2] dlopen (repeats 2 times)
    @ ./libdl.jl:116 [inlined]
  [3] load_libs(always::Bool)
    @ GR ~/.julia/dev/GR/src/funcptrs.jl:33
  [4] init(always::Bool)
    @ GR ~/.julia/dev/GR/src/GR.jl:298
  [5] init
    @ ~/.julia/dev/GR/src/GR.jl:297 [inlined]
  [6] plot_data(flag::Bool, plt::GR.jlgr.PlotObject)
    @ GR.jlgr ~/.julia/dev/GR/src/jlgr.jl:1180
  [7] plot_data (repeats 2 times)
    @ ~/.julia/dev/GR/src/jlgr.jl:1176 [inlined]
  [8] plot(::Vector{Float64}, ::Vararg{Vector{Float64}}; kv::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ GR.jlgr ~/.julia/dev/GR/src/jlgr.jl:1746
  [9] plot(::Vector{Float64}, ::Vararg{Vector{Float64}})
    @ GR.jlgr ~/.julia/dev/GR/src/jlgr.jl:1737
 [10] top-level scope
    @ REPL[7]:1

julia> GR.GRPreferences.diagnostics()
┌ Info: GRDIR Environment Variable
└   get(ENV, "GRDIR", missing) = "/tmp/tmp.mFM5iDWhYr/newgr/gr"
┌ Info: GR Preferences
│   binary = "system"
└   grdir = "/tmp/tmp.mFM5iDWhYr/newgr/gr"
┌ Info: resolved_grdir
│   isdir(resolved_grdir) = true
└   isdir.(joinpath.((resolved_grdir,), ("bin", "lib", "include", "fonts"))) = (true, true, true, true)
┌ Info: GR_jll Preferences
│   libGR_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGR"
│   libGR3_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGR3"
│   libGRM_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGRM"
│   libGKS_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGKS"
└   gksqt_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/bin/gksqt"
┌ Info: GR_jll Override.toml
│   override_toml_path = "/home/mkitti/.julia/artifacts/Overrides.toml"
│   isfile(override_toml_path) = true
│   gr_jll_override_dict =
│    Dict{String, Any} with 1 entry:
└      "GR" => "/tmp/tmp.mFM5iDWhYr/newgr/gr"
┌ Info: GR_jll
│   GR_jll.libGR_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR"
│   GR_jll.libGR3_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGR3"
│   GR_jll.libGRM_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGRM"
│   GR_jll.libGKS_path = "/tmp/tmp.mFM5iDWhYr/gr/lib/libGKS"
└   GR_jll.gksqt_path = "/tmp/tmp.mFM5iDWhYr/gr/bin/gksqt"
(binary = "system", grdir = "/tmp/tmp.mFM5iDWhYr/newgr/gr", libGR_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGR", libGR3_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGR3", libGRM_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGRM", libGKS_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/lib/libGKS", gksqt_path = "/tmp/tmp.mFM5iDWhYr/newgr/gr/bin/gksqt", override_toml_path = "/home/mkitti/.julia/artifacts/Overrides.toml", gr_jll_override_dict = Dict{String, Any}("GR" => "/tmp/tmp.mFM5iDWhYr/newgr/gr"))

julia> GR.GRPreferences.__init__()
"/tmp/tmp.mFM5iDWhYr/newgr/gr/lib"

julia> plot(x,y)
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.

mkitti · 2022-11-02T20:50:09Z

@giordano I'm good to merge this. I'll look into the PATH issue on Windows.

mkitti · 2022-11-08T14:29:01Z

For the record this was yanked, since we need a few lines in GR.jl for this to work:
https://github.com/jheinen/GR.jl/blob/dd3913dc824af287151c22b4ffc29f3df6b0ed54/src/preferences.jl#L61-L65

Use dont_dlopen=true with GR libraries

1e6e8c8

We're probably going to need a version bump for this. Alternatively, I'm looking into using Overrides.toml

mkitti mentioned this pull request Nov 2, 2022

Return of the build script via Preferences jheinen/GR.jl#483

Merged

giordano merged commit bb18b14 into JuliaPackaging:master Nov 2, 2022

This was referenced Nov 3, 2022

GKS: svgplugin.dll: can't load library, error 126 (0x7e) jheinen/GR.jl#486

Closed

Synchronize with build_tarballs.jl in Yggdrasil jheinen/GR.jl#484

Closed

This was referenced Nov 3, 2022

GKS: svgplugin.dll: can't load library, error 126 (0x7e) jheinen/GR.jl#487

Closed

Exact pin of GR_jll jheinen/GR.jl#488

Closed

mkitti mentioned this pull request Nov 4, 2022

Using GR plugins without setting PATH on Windows jheinen/GR.jl#489

Closed

mkitti mentioned this pull request Nov 8, 2022

Path to GRDIR environment variable configuration jheinen/GR.jl#491

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use dont_dlopen=true with GR libraries #5784

Use dont_dlopen=true with GR libraries #5784

mkitti commented Nov 1, 2022 •

edited

Loading

giordano commented Nov 1, 2022

jheinen commented Nov 2, 2022 •

edited

Loading

mkitti commented Nov 2, 2022

vchuravy commented Nov 2, 2022

sjkelly commented Nov 2, 2022 via email

t-bltg commented Nov 2, 2022 •

edited

Loading

mkitti commented Nov 2, 2022

t-bltg commented Nov 2, 2022

mkitti commented Nov 2, 2022 •

edited

Loading

sjkelly commented Nov 2, 2022 •

edited

Loading

mkitti commented Nov 2, 2022

mkitti commented Nov 2, 2022

jheinen commented Nov 2, 2022 •

edited

Loading

mkitti commented Nov 2, 2022

mkitti commented Nov 2, 2022

giordano commented Nov 2, 2022 •

edited

Loading

giordano commented Nov 2, 2022

t-bltg commented Nov 2, 2022 •

edited

Loading

mkitti commented Nov 2, 2022

mkitti commented Nov 2, 2022

mkitti commented Nov 8, 2022

Use dont_dlopen=true with GR libraries #5784

Use dont_dlopen=true with GR libraries #5784

Conversation

mkitti commented Nov 1, 2022 • edited Loading

giordano commented Nov 1, 2022

jheinen commented Nov 2, 2022 • edited Loading

mkitti commented Nov 2, 2022

vchuravy commented Nov 2, 2022

sjkelly commented Nov 2, 2022 via email

t-bltg commented Nov 2, 2022 • edited Loading

mkitti commented Nov 2, 2022

t-bltg commented Nov 2, 2022

mkitti commented Nov 2, 2022 • edited Loading

sjkelly commented Nov 2, 2022 • edited Loading

mkitti commented Nov 2, 2022

mkitti commented Nov 2, 2022

jheinen commented Nov 2, 2022 • edited Loading

mkitti commented Nov 2, 2022

mkitti commented Nov 2, 2022

giordano commented Nov 2, 2022 • edited Loading

giordano commented Nov 2, 2022

t-bltg commented Nov 2, 2022 • edited Loading

mkitti commented Nov 2, 2022

mkitti commented Nov 2, 2022

mkitti commented Nov 8, 2022

mkitti commented Nov 1, 2022 •

edited

Loading

jheinen commented Nov 2, 2022 •

edited

Loading

t-bltg commented Nov 2, 2022 •

edited

Loading

mkitti commented Nov 2, 2022 •

edited

Loading

sjkelly commented Nov 2, 2022 •

edited

Loading

jheinen commented Nov 2, 2022 •

edited

Loading

giordano commented Nov 2, 2022 •

edited

Loading

t-bltg commented Nov 2, 2022 •

edited

Loading