binarybuilder_lowerbound_discussion.txt

Chris Foster  6 days ago
But I guess we currently need to supply things like Zlib_jll in the Project.toml [deps] section?
Mosè Giordano  6 days ago
Yes, exactly. The way the jll packages work doesn't allow them to be installed in julia <1.3, I don't see many options at the moment to make BinaryProvider and jll coexist in those versions of julia
Chris Foster  6 days ago
I see one possible option: the jll packages themselves could allow installation on julia < 1.3 and just load as an empty module in that case. That would allow some recourse for maintainers who have to support older julia versions.
Fredrik Ekre  6 days ago

    just load as an empty module in that case

packages could just do something like

if VERSION >= v"1.3"
    using Zlib_jll
end

Chris Foster  6 days ago
That's not the problem; the issue is with the Project.toml
Fredrik Ekre  6 days ago
I mean, if we allow them to be installed then that is not a problem
Chris Foster  6 days ago
Oh right. Yes true.
Chris Foster  6 days ago
That sounds pretty good actually. It's opt-in for the package which depends on the jll and that seems like the right way around.
Mosè Giordano  6 days ago
I seem to remember that there was a problem with using in conditionals (like if false; using Foo; end would load Foo anyway), is this the case?
Fredrik Ekre  5 days ago
Thats what Compat used to do with the moved stdlibs, so I think it works.
Mosè Giordano  5 days ago
I always thought (but never actually tried) that there was this problem with conditionally loading packages, but if it's not then this should work. you only need to convince Elliot to update all JLL packages :stuck_out_tongue:
Fredrik Ekre  5 days ago
I have the power to update the bounds in the registry, so I don't need Elliot :sunglasses:
Mosè Giordano  5 days ago
the packages require 1.3 in Project.toml.  doesn't that need to be changed?
Fredrik Ekre  5 days ago
No :stuck_out_tongue:
Mosè Giordano  5 days ago
how does that work?  is that requirement ignored? or the compat entry in the registry has precedence? I'm confused
Fredrik Ekre  5 days ago
The actual project file is only read when registering a package.
Fredrik Ekre  5 days ago
For regular version resolution only the registry is used, so we can just modify the compat bounds in the registry
Mosè Giordano  5 days ago
oh, I see.  is Project.toml used for unregistered packages? (edited) 
Fredrik Ekre  5 days ago
Yea, when the package is in a checked-out state it is used.
Mosè Giordano  5 days ago
ok, thanks
Chris Foster  5 days ago
Yeah, you wouldn't be able to dev the jll packages and have them work properly unless the Project.toml is updated. But also that's an unlikely thing to do if they're all autogeneraed. (edited) 
David Anthoff  5 days ago
Even crazier idea: the jll packages could have a build.jl that only runs on julia <1.3, and uses BinaryProvider to actually install the binaries
Elliot Saba:horse:  5 days ago
I'm pretty :neutral_face: about having JLL packages installed on your system that will error if you try to using them.  I think this is a good opportunity to force users to upgrade to 1.3 to get the goodies we've been working on for them.  If you want to support 1.2-, I think you should generate build.jl files and continue to use BinaryProvider, that's fine and will mostly continue to work (packages like Gtk can work with BP, it just requires a lot of fiddling to get everything in the right order, etc....).
Elliot Saba:horse:  5 days ago
I've already considered having BinaryProvider as an automatic backup for Julia 1.2-, but since the feature sets are not 1:1 we will still end up with the situation where users ask "why doesn't this work on our machine" and the answer will be "you need to upgrade to Julia 1.3 to get the new goodies"
Elliot Saba:horse:  5 days ago
So in the end, I really do think pushing people to upgrade is the way to go, unless there is a strong reason not to (e.g. for a really big package like Gtk, where the chances that people need it and can't upgrade are very high
Mosè Giordano  5 days ago
I was expecting this reaction :smile:
David Anthoff  5 days ago
Me too :slightly_smiling_face:
Chris Foster  5 days ago
I don't think it's helpful to completely prevent package authors from gracefully transitioning to the new system. There is still going to be plenty of incentive to upgrade, as the new system makes binary deps so much easier that there will be plenty of new 1.3-only packages.
Chris Foster  5 days ago
If we don't allow jll's to be installable on earlier julia versions, we're making the choice for maintainers who really care about supporting older versions that they should stick with build.jl for now.
Elliot Saba:horse:  5 days ago
Yes, that’s correct. Once they are ready to branch off and release 1.3+ versions of their packages, they’ll be able to use the new stuff. It seems to me that many packages are going to be doing this soon, as the multithreading capabilities are going to be pretty transformative to the ecosystem, and are similarly gated on the version of Julia.
Chris Foster  5 days ago
But I feel there's a fundamental downside to this reasoning!
You are forcing an unnecessary coupling between package development progress (which, with good collaboration, can be cross-organizational) to project deployment considerations which are unique to each organization.
So if a company's current project deployment environment says that they're using julia < 1.3, any jll-using PRs to that organization's repos will be blocked until they decide to upgrade. It's simply not in their interests to merge these things until they're also on the 1.3 bandwagon.
Chris Foster  5 days ago
This means that other users outside the company will also be blocked from upgrading to jll dependencies as their PRs will be held up. It's not the end of the world, but it's bad for collaboration on any package which is still in heavy development and which has users in different deployment environments. (edited) 
Simon Byrne:juliaspinner:  5 days ago
I agree, we are going to effectively bifurcate all package versions
Simon Byrne:juliaspinner:  5 days ago
which is not ideal
Elliot Saba:horse:  5 days ago

     So if a company's current project deployment environment says that they're using julia < 1.3, any jll-using PRs to that organization's repos will be blocked until they decide to upgrade. 

This feels natural to me, because the features that package would be using are not in Julia <1.3.  I think there may be some misconceptions about what a "JLL package" actually provides; I'm not saying "we shouldn't provide a nice API on older versions of Julia because I want to pressure people to upgrade".  Instead I'm saying "these are really nice features that are only available on newer versions of Julia, so it's good that it draws them forwards".Technically, we could do as suggested above, and have loading a JLL package be a no-op on Julia <1.2.  The question becomes, what is the true utility of this for the user?  Can we actually provide feature-equivalence on Julia 1.2 by using build.jl files such that they will be able to get all the good stuff while using 1.3, but have a reasonable fallback when using 1.2?

    JLL packages work with the resolver, enforcing a single version of a dependency (across a single environment) is used everywhere.  build.jl files, in their most common form, install the entire tree of dependencies into each Julia package folder; this will work as long as all packages across the entire Julia ecosystem stay in lockstep with eachother (e.g. the version of Fontconfig  being installed by Gtk.jl and Cairo.jl must be the same, otherwise running using Gtk  will work, but running using Cairo, Gtk  will not.
    JLL packages provide an API for doing things like calling executables with the particular  PATH and LD_LIBRARY_PATH environment variables needed to find all their dependencies.  If everything a package (and all its recursive dependencies) needs is installed into its own deps folder, setting up these environment variables is not that difficult, but if you don't want to re-download all these dependencies (And you want to mimic the 1.3+ syntax), you'll have to generate those kinds of wrappers as well.  This is not hard, but as we continue to add new capabilities to JLL packages on 1.3+, we'll have to continue to find ways to intelligently emulate this behavior on the build.jl-based system, and there may not always be a graceful way to do that.  Example: JLL packages guarantee that all libraries are dlopen()'ed by the time __init__() is finished within the JLL package, and we're in the midst of allowing JLL packages to declare flags that should be used to dlopen those libraries to enable strange situations like a library needing another library loaded with RTLD_GLOBAL first before it can even be loaded.  These kinds of situations are, of course, able to be hacked around, but it all takes developer effort and will not be automatic on 1.2.
    The Artifacts system is a big step forward for the Julia ecosystem towards making things relocatable; as such, packages that need to write data out to disk (such as Gtk, as it needs to generate the gdk-pixbuf loader cache) are being encouraged to use the Artifacts system to store those kinds of data to disk so that the package directories can remain relocatable and more importantly, read-only.  Again, this is hack-around-able, but it's just another example of features that exist only in 1.3 and not in earlier versions.

    I agree, we are going to effectively bifurcate all package versions

Elliot Saba:horse:  5 days ago
This is simply the nature of a new version of Julia; if we were discussing @spawn I don't think anyone would be surprised if I said "Code that uses @spawn will do fundamentally different things on 1.2".  This is a good analogy because @spawn does do something on 1.2, but what it does is fundamentally different, and code that is written for the 1.3 meaning of @spawn may not work at all when run on 1.2.  Are we going to bifurcate all packages that use parallel codes?  Yes.  I am already hearing about data loading packages (for things like DataFrames) that are adding multithreaded loading and parsing in because it's a big win for performance, and that new version of the package will naturally only work on 1.3+.  Similarly, I fully expect Gtk.jl to switch to a 1.3+ only release soon, with a separate branch for 1.2-.  This happens every time we make a release, and it's a good thing.  The fact that we're adding fundamentally new capabilities that can't be easily emulated on older versions of Julia is a good thing.  Otherwise, we may as well just call this a new entry in Julia 1.2.X and backport everything.
Elliot Saba:horse:  5 days ago
Wow, slack split my giant message up into multiple posts.  That's interesting.
Chris Foster  5 days ago
We're just talking about different things here.
Chris Foster  5 days ago
I'd like to suggest that @fredrikekre's approach is ideal: jll packages specify julia 1.0 as a requirement in Project.toml, but just fail to load on julia < 1.3.
Chris Foster  5 days ago
To illustrate my point with a story.
Let's consider LibFoo.jl version 1.0 is released and is currently maintained/controlled by CompanyA. It's open source, but it's also used in an important ProductionSystem which is internal to CompanyA.
Let's suppose that CompanyA has done some ugly hackery to arrange for C binary libfoo to be available in production. (This is not hypothetical, but a typical situtation in the pre BinaryBuilder days.)
Let's also suppose CompanyA won't commit dev time to upgrade ProductionSystem to julia 1.3 until 2021, but that they continue to improve the completeness of LibFoo's C bindings and add features, thus upgrading through semver 1.1, 1.2 1.3...
Chris Foster  5 days ago
Now OpenSourceDevB comes along and wants to improve LibFoo.jl  to make it usable on julia 1.3 with all the goodness of having consistent libs across the board.
OpenSourceDevB does the work and submits a PR, which necessarily adds libfoo_jll to the [deps] section of the Project.toml
Chris Foster  5 days ago
Now what happens?
CompanyA's devs would love to merge this PR, but they also want to release version 1.4 of LibFoo.jl  next month and use it in their production system.
Therefore, they postpone merging the PR until 2021.
Chris Foster  5 days ago
There seems to be no way for CompanyA and OpenSourceDevB to collaborate because there can only be one Project.toml for LibFoo.jl version 1.4 and the presence or absence of libfoo_jll is mutually exclusive between versions. (edited) 
Chris Foster  5 days ago
Note that this observation is completely independent of whatever fancy features julia-1.3 brings to the table.
For @spawn, package developers have a choice: they can choose to use of this feature behind an if VERSION >= v"1.3" check if compatibility really matters to them. (edited) 
Chris Foster  5 days ago
I'd say that incompatibility in Project.toml forces package developers to make exactly the uneasy choice which plagued the python 2->3 transition: Stay on the earlier version for compatibility, or transition to the new version for features. But "thou shalt not support both sets of your users from the same codebase".
Elliot Saba:horse:  4 days ago
I think what is lacking here is an understanding of how allowing libfoo_jll to silently fail on 1.2 would help things; can you explain how you intend to deal with the differences between JLL packages/yes artifacts and build.jl files/no artifacts within the same package?  From my perspective, they have some pretty big differences in capability, such that it does not make sense to pretend that you could write a Julia package that makes use of both of them.  (In the general case of course; there are simple examples where they are all but identical and this will not be a problem)
Elliot Saba:horse:  4 days ago
What I have been trying to communicate is that there are significant enough differences in what Julia 1.3+ is trying to do that it will actually end up changing the way you write packages (such as creating mutable artifacts to cache local setup), and if you were to keep your package written in such a way that using a build.jl file was an option at all, you might as well not use JLL packages at all.  That's a bit of an overstatement, but it will only become more true as time goes on.
Chris Foster  4 days ago
Thanks Elliot I do appreciate your explanation of your point of view, even if I still disagree with it :slightly_smiling_face:
Let me summarize how I imagine the new system works since it's very new to me. Perhaps I have some misunderstanding there.
Chris Foster  4 days ago
(back shortly... need to take the kids to school)
Chris Foster  4 days ago
So, let's consider libpng_jll . From the Pkg resolver point of view, it depends on Zlib_jll (and on stdlibs Pkg, Libdl) as described in the Project.toml.
So the resolver will make sure a compatible version of Zlib_jll is installed if you add libpng_jll.
So far all the same as in julia < 1.3 from a user's perspective, with the addition of an extra "oddly named" dependency.When libpng_jll is loaded, the artifacts system is used to find(/download?) the correct platform-specific binary.
The platform-specific wrappers then handle path munging and dlopening of the underlying libpng library.
Chris Foster  4 days ago
The effect of using the resolver is to correctly deal with complicated interdependent versioning constraints between binaries.
I think this part could have been done in 1.0 if there was a community-wide effort to build compatible binaries and to correctly represent the version dependency graph?
Chris Foster  4 days ago
What's new is the artifacts system which allows the package to remove the build.jl which would otherwise be required to mutate the package directory.
Elliot Saba:horse:  4 days ago

    So far all the same as in julia < 1.3 from a user's perspective, with the addition of an extra "oddly named" dependency.

This depends on what you mean by "from a user's perspective"; if by that you mean "ignoring the technical details" then yes, it looks very similar (and that is a valid perspective).  But there are really important subtleties here.Going back to my example using Cairo and Gtk above, because the binaries are installed into e.g. Cairo/deps and Gtk/deps, and they have no real way to communicate the paths of the binary dependencies to eachother, Gtk installs a duplicate libcairo into Gtk/deps so that libgtk.so can find it at dlopen() time. Thus, Gtk must coordinate binary versions with Cairo , so that you don't have libcairo.so.1.14 in Cairo/deps, and libcairo.so.1.16 in Gtk/deps.  If you have that, using Cairo will work, and using Gtk will work, but using Cairo, Gtk won't work.Allowing the resolver to know about things enables us to install a single version of libcairo for both Gtk and Cairo.  Introducing JLL packages allows us to jump through the hoops the dynamic linker requires in order to force libgtk to find libcairo at dlopen() time (since their relative positioning is completely unknown)

    The effect of using the resolver is to correctly deal with complicated interdependent versioning constraints between binaries.

This is true, but it's not the whole truth.  The most important thing is to resolve down to a single version at a time, instead of having multiple versions getting loaded in to the process address space.
Elliot Saba:horse:  4 days ago

    I think this part could have been done in 1.0 if there was a community-wide effort to build compatible binaries and to correctly represent the version dependency graph?

You're technically correct in that it is possible (in the absolute sense) to have a "userspace" implementation of this, where packages coordinate by exporting paths to libcairo.so, Gtk would use Cairo.libcairo, and would manipulate things like DYLD_LIBRARY_PATH and whatnot to get dlopen() to be happy.  That is possible and it does not rely on inherent compiler tech that we don't have in 1.0
Elliot Saba:horse:  4 days ago
That's essentially what JLL packages do; they provide all that extra glue code that is boring and finnicky.  And yes, with the exception of using Pkg.Artifacts to do the downloading, it is totally technically possible to get these benefits on Julia 1.0
Elliot Saba:horse:  4 days ago
So you could, for instance, write JLL packages that have build.jl files that download things at runtime, and do that instead of using artifacts on 1.3.  You would lose some advantages, but you would keep the advantages listed above (including the "only one version of a library at a time" advantage)
Chris Foster  4 days ago
Going further, it would even be "possible" to have a compatibility package, PkgArtifacts.jl which would bring the centralized Pkg.Artifacts cache to older julia versions? I ask in principle; not because I expect anyone to actually do the work ;-) (edited) 
Elliot Saba:horse:  4 days ago
Yes; that's correct.  But you've correctly identified that there's not much interest in doing all of that work.  :wink:
Chris Foster  4 days ago
Ok, so I think I do understand the shape of the problem and solution (and I've read several blah_jll packages in the process)
Chris Foster  4 days ago

    What I have been trying to communicate is that there are significant 
    enough differences in what Julia 1.3+ is trying to do that it will 
    actually end up changing the way you write packages (such as creating 
    mutable artifacts to cache local setup),

Sure, fair enough!

    if you were to keep your 
    package written in such a way that using a build.jl
     file was an option at all, you might as well not use JLL packages at 
    all.  That's a bit of an overstatement, but it will only become more 
    true as time goes on.

This is what I strongly disagree with. I think we should allow for a transition period where packages are permitted to use the existing hacks in build.jl  on julia < 1.3, while also supporting the awesome new 1.3 way of doing things (gated on a VERSION check internal to the package).
Everything you've explained to me seems like implementation detail of the package which should not affect the user-visible API of the package at all. (edited) 
Chris Foster  4 days ago
It should be the package developer's choice to decide whether they want to go through the pain of supporting both mechanisms.
Fundamentally this is about empowering the package developers to make the choice necessary for their deployment environments rather than trying to make that choice for them.
Chris Foster  4 days ago
By the way, I don't think there's a need to add artificial incentives to upgrade julia versions.
The 1.3 artifacts+binary system is so compelling that anyone struggling with deployment will see its inherent benefits and want to upgrade.
Chris Foster  4 days ago
But for complex production systems this just takes time and effort. Allowing package developers to support a smooth transition (at their own effort!) seems as simple as changing julia = 1.3 to julia = 1.0 in the jll Project.toml files. I just can't seen any downside to this. (edited)