Gentoo Forums
Gentoo Forums
Gentoo Forums
Quick Search: in
[CLOSED] Why do we have dependency hell?
View unanswered posts
View posts from last 24 hours

 
Reply to topic    Gentoo Forums Forum Index Portage & Programming
View previous topic :: View next topic  
Author Message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Thu Jun 25, 2020 7:10 pm    Post subject: [CLOSED] Why do we have dependency hell? Reply with quote

Of all I thought we shouldn't have such a problem.

Since everything is compiled and everything is versioned, it should be possible to keep all versions currently in use, and link with the needed one (right?).


Last edited by extowgen on Fri Jun 26, 2020 10:00 pm; edited 1 time in total
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45826
Location: 56N 3W

PostPosted: Thu Jun 25, 2020 7:16 pm    Post subject: Reply with quote

extowgen,

Close.
Not all versions of everything can be installed at the same time.

If you run a stable system, you dependency issues are either bugs or a problem of your own making.
Its like that with all ~arch too but its expected to be a bit rough round the edges.

When you mix stable and testing, you get to resolve the issues yourself.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Ionen
l33t
l33t


Joined: 06 Dec 2018
Posts: 889

PostPosted: Thu Jun 25, 2020 7:27 pm    Post subject: Reply with quote

Given things sometime need different versions of the same package that would require every package versions to be slotted which is not trivial, and then everything that use those packages would need to be able to find them with their new naming/location that is needed when using slots. Not a big issue when dealing with 1 package, but hundreds/thousands with their own issues and build systems not so much.

Like if you need two version of bash, there's only one that will be at /bin/bash. And if a package has a script that need a old version of it due to poor writing you'd need to replace all its #!/bin/bash to be #!/bin/bash-version or some weirdo bash-exec selector (trivial but just an example). The more normal solution would be to hold back updating bash until that package is fixed.. and if there's no maintainer/dead-upstream it's probably time to use something else or fix it yourself :)
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Thu Jun 25, 2020 7:35 pm    Post subject: Reply with quote

Ionen wrote:
Given things sometime need different versions of the same package that would require every package versions to be slotted which is not trivial, and then everything that use those packages would need to be able to find them with their new naming/location that is needed when using slots. Not a big issue when dealing with 1 package, but hundreds/thousands with their own issues and build systems not so much.

Like if you need two version of bash, there's only one that will be at /bin/bash. And if a package has a script that need a old version of it due to poor writing you'd need to replace all its #!/bin/bash to be #!/bin/bash-version or some weirdo bash-exec selector (trivial but just an example). The more normal solution would be to hold back updating bash until that package is fixed.. and if there's no maintainer/dead-upstream it's probably time to use something else or fix it yourself :)


well everything has a version, so let all binaries and static libs have a version too, thus there should be no problem.

If you as a user need only one specific version but don't want to type the version or you don't care which version, you create a link.
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Thu Jun 25, 2020 7:36 pm    Post subject: Reply with quote

NeddySeagoon wrote:
extowgen,

Close.
Not all versions of everything can be installed at the same time.

If you run a stable system, you dependency issues are either bugs or a problem of your own making.
Its like that with all ~arch too but its expected to be a bit rough round the edges.

When you mix stable and testing, you get to resolve the issues yourself.


why can't all needed versions be installed at the same time?
Back to top
View user's profile Send private message
Ionen
l33t
l33t


Joined: 06 Dec 2018
Posts: 889

PostPosted: Thu Jun 25, 2020 7:46 pm    Post subject: Reply with quote

If anything this is easier to accomplish on a binary distribution, it's a similar idea to flatpak and the like that have all their dependencies included and build-time dependencies are no longer relevant, and then none of this matters (until the dependency have security flaws or break with a given kernel anyway).

Gentoo is intended to be a single system that works together rather than everything being in their own chroot/VM and the like with their self-contained dependencies not to conflict.
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Thu Jun 25, 2020 8:08 pm    Post subject: Reply with quote

Ionen wrote:
If anything this is easier to accomplish on a binary distribution, it's a similar idea to flatpak and the like that have all their dependencies included and build-time dependencies are no longer relevant, and then none of this matters (until the dependency have security flaws or break with a given kernel anyway).

Gentoo is intended to be a single system that works together rather than everything being in their own chroot/VM and the like with their self-contained dependencies not to conflict.


I'm not saying anything about being self-contained. Just have all needed dependency versions installed, and each package already knows which version of each dependency it needs, the only thing missing is to propagate the versions to the actual build, so when linking a shared library it specifies the version instead of leaving it out, and the like.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45826
Location: 56N 3W

PostPosted: Thu Jun 25, 2020 8:14 pm    Post subject: Reply with quote

extowgen,

Code:
why can't all needed versions be installed at the same time?

Different versions tend to use identical file names for things.

To prevent file name collisions files would need to be renamed on install.
Worse, packages that link to the renamed files need to know the new names.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
GDH-gentoo
Guru
Guru


Joined: 20 Jul 2019
Posts: 434
Location: South America

PostPosted: Thu Jun 25, 2020 8:15 pm    Post subject: Reply with quote

extowgen wrote:
why can't all needed versions be installed at the same time?
Filename collision. Packages that do allow having multiple versions installed (implemented in Gentoo via slots) can exist when this can somehow be avoided with a reasonable amount of packaging tricks. And possibly, only when it is needed or convenient: see next answer.

extowgen wrote:
well everything has a version, so let all binaries and static libs have a version too, thus there should be no problem.
In filenames you mean? And with the number of packages that a typical 'big' distribution offers, and the number of files that each package contains, who is going to do the work of renaming each of them? It's just too much to be feasible.

EDIT to add: And also, programs that use one of the exec...() calls, or the dlopen() call, contain filenames in their code.

Ionen wrote:
Gentoo is intended to be a single system that works together rather than everything being in their own chroot/VM and the like with their self-contained dependencies not to conflict.
I believe most distributions are intended to be a single system like Gentoo, in fact. The versioned binary-based ones just don't have this problem that much because each release offers a single version of every package, but they have others. You want a more recent one? You wait for the next release of the distribution. Version X of package Y has a serious bug or vulnerability that is fixed in version Z? You wait until the distribution's developers backport the fixes to version X.

I'd expect rolling release binary-based distributions to have more or less the same "dependency hell" that Gentoo has. Because it is likely just a reflection of upstream hell.


Last edited by GDH-gentoo on Thu Jun 25, 2020 8:38 pm; edited 5 times in total
Back to top
View user's profile Send private message
Ionen
l33t
l33t


Joined: 06 Dec 2018
Posts: 889

PostPosted: Thu Jun 25, 2020 8:24 pm    Post subject: Reply with quote

It's not that it's _impossible_ but as I said earlier, while you can set this up for a single package relatively easily, for an entire distribution it becomes quite a ridiculous task given the amount of packages to deal with. It doesn't just magically work as-is and would preferably need to be a design goal from the beginning with a workforce willing to put in the additional work (gentoo would end up quite different too, not sure I'd even want to use that given it'd require far more non-standard workarounds that cause more incompatibility issues for anything not managed by the distro).
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Thu Jun 25, 2020 8:54 pm    Post subject: Reply with quote

GDH-gentoo wrote:

EDIT to add: And also, programs that use one of the exec...() calls, or the dlopen() call, contain filenames in their code.


that can be fixed with the help of the OS. But I guess no one would dare to do it.

Ionen wrote:
It's not that it's _impossible_ but as I said earlier, while you can set this up for a single package relatively easily, for an entire distribution it becomes quite a ridiculous task given the amount of packages to deal with. It doesn't just magically work as-is and would preferably need to be a design goal from the beginning with a workforce willing to put in the additional work (gentoo would end up quite different too, not sure I'd even want to use that given it'd require far more non-standard workarounds that cause more incompatibility issues for anything not managed by the distro).


well,l it should have been done from the beginning.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45826
Location: 56N 3W

PostPosted: Thu Jun 25, 2020 8:56 pm    Post subject: Reply with quote

extowgen,

Patches welcome.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15623

PostPosted: Fri Jun 26, 2020 2:14 am    Post subject: Reply with quote

extowgen wrote:
GDH-gentoo wrote:
EDIT to add: And also, programs that use one of the exec...() calls, or the dlopen() call, contain filenames in their code.
that can be fixed with the help of the OS. But I guess no one would dare to do it.
In the abstract, every software problem can be fixed with more software. How specifically do you propose to solve this? For your chosen solution, which developers are responsible for it, how much extra effort will it add to them, and how much ongoing maintenance will it require?
extowgen wrote:
Ionen wrote:
It's not that it's _impossible_ but as I said earlier, while you can set this up for a single package relatively easily, for an entire distribution it becomes quite a ridiculous task given the amount of packages to deal with. It doesn't just magically work as-is and would preferably need to be a design goal from the beginning with a workforce willing to put in the additional work (gentoo would end up quite different too, not sure I'd even want to use that given it'd require far more non-standard workarounds that cause more incompatibility issues for anything not managed by the distro).
well,l it should have been done from the beginning.
There are a lot of things that should have been done better long ago. People back then either didn't know those things were needed, or made a judgment call that the things were not needed enough to be worth it. For example:
  • CPUs should have had NX support when they first introduced MMUs, not years later.
  • CPUs should have treated R/W/X permissions separately, so that code pages could be --X instead of R-X.
  • Security-sensitive data should not be in the same memory block as untrusted data.
  • Low level languages should have had the ability to pass more size/type information, so that static analysis could be more advanced now than it is.
  • Every non-trivial language should have shipped with a bidirectional language <-> AST converter, and convention would be that source controlled objects are ASTs, not the pretty text. This would make many forms of refactoring and code analysis much easier.
  • We should have had user namespaces 20 years ago, and all the cool toys they bring.
  • Hardware should have shipped with useful data sheets as standard for the last 30 years, so we could have had working free drivers for more hardware much sooner.
  • Vendors should ship a reasonable driver concurrent with launching their hardware, and should make an effort to get it into the core kernel.
  • Encryption and authentication should have been standard in every protocol, rather than haphazardly added on via TLS for some things and not others, and only when both sides happen to support it.
I could go on for a long time lamenting things that we have now that we should have had then, and if I get to list things we still don't have, but that I wish we did, the list gets even longer. The common thread in every case is that the people who would have done the work either didn't know it was important, or didn't think it was important enough relative to the other work they could be doing.
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Fri Jun 26, 2020 9:05 am    Post subject: Reply with quote

Hu wrote:
extowgen wrote:
GDH-gentoo wrote:
EDIT to add: And also, programs that use one of the exec...() calls, or the dlopen() call, contain filenames in their code.
that can be fixed with the help of the OS. But I guess no one would dare to do it.
In the abstract, every software problem can be fixed with more software. How specifically do you propose to solve this? For your chosen solution, which developers are responsible for it, how much extra effort will it add to them, and how much ongoing maintenance will it require?

(Here by a path without a version I mean a path to a link, since every binary would be versioned)

When a process tries to create another process or dynamically load a library without specifying the version, the OS checks whether this binary file is one of the caller's dependencies, if so, it redirects the call to an appropriate version.

I don't know whether there's a practice to specify an absolute path to a binary and expect a specific version. If there is, the user would need to specify(in some config of the OS, or by a command) which directories have links [to binaries] that should not be redirected. Then the call with any path without a version not located in the aforementioned directories and is the caller's dependency would be redirected to an appropriate version.

One downside: for programs that execute commands given by the user, the user would have to specify a path to a link located in the aforementioned directory, or simply specify the version. But the good thing is that kind of programs don't have many dependencies which the user would want to execute.

Hu wrote:
There are a lot of things that should have been done better long ago. People back then either didn't know those things were needed, or made a judgment call that the things were not needed enough to be worth it.


Some of the things you've mentioned are quite subtle and require a lot of thought and practice.

The problem with dependencies is just in your face. If you decide to have everything shared, then you immediately face the problem.
It gets under my skin, that I can't have installed something old and something new. Like what happens if a given package is not maintained anymore? A bunch of nonsense happens.
Microsoft figured it's better not to overengineer, so the users of Windows can't even imagine that such a problem exists.
Back to top
View user's profile Send private message
NeddySeagoon
Administrator
Administrator


Joined: 05 Jul 2003
Posts: 45826
Location: 56N 3W

PostPosted: Fri Jun 26, 2020 3:26 pm    Post subject: Reply with quote

extowgen,

Its been a while since I had to admin windows. Its nearly 20 years since I suffered from it at home.
I well remember *.dll hell.
_________________
Regards,

NeddySeagoon

Computer users fall into two groups:-
those that do backups
those that have never had a hard drive fail.
Back to top
View user's profile Send private message
GDH-gentoo
Guru
Guru


Joined: 20 Jul 2019
Posts: 434
Location: South America

PostPosted: Fri Jun 26, 2020 4:43 pm    Post subject: Reply with quote

extowgen wrote:
When a process tries to create another process or dynamically load a library without specifying the version, the OS checks whether this binary file is one of the caller's dependencies, if so, it redirects the call to an appropriate version.
"The OS" is a very generic term in this context. Let's get a bit more detailed. exec...() is ultimately implemented as a Linux system call. dlopen() is implemented by the libc. Dependencies are recorded by the package manager. Are you suggesting that kernel developers, GNU libc developers, musl developers, etc. should change their software to have it consult the package manager? Which one? APT, DNF, Portage, Pacman, apk, XBPS, ...? All of them? How do you propose the kernel and libc determine the identity of "the caller", so that they can then determine which package the caller belongs to, and perform the dependency query?

extowgen wrote:
I don't know whether there's a practice to specify an absolute path to a binary and expect a specific version. If there is, the user would need to specify(in some config of the OS, or by a command) which directories have links [to binaries] that should not be redirected. Then the call with any path without a version not located in the aforementioned directories and is the caller's dependency would be redirected to an appropriate version.
So, in addition to interfacing some package manager, should kernel developers also include a configuration file parser as part of exec...()'s implementation? Should /usr/bin, for example, be a directory with "redirections to appropriate versions" behaviour, or not?

Also, you haven't addressed the workload aspect. What's your estimation of the man-hours needed to migrate all packages to your versions-in-filenames scheme? Who does the work? Distributions? Upstreams?
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Fri Jun 26, 2020 9:57 pm    Post subject: Reply with quote

Ok, now that my mind has cleared(mainly thanks to the captious questions of everyone involved), I can see there's an inherent problem in every OS, yet the problem is not due to a poor design choice, but rather due to human laziness.

Thank you for your patience.
Back to top
View user's profile Send private message
Ant P.
Watchman
Watchman


Joined: 18 Apr 2009
Posts: 6637

PostPosted: Fri Jun 26, 2020 11:09 pm    Post subject: Reply with quote

I would prefer updating one dynamic library over risking fifteen bundled copies of Electron.
Back to top
View user's profile Send private message
Hu
Moderator
Moderator


Joined: 06 Mar 2007
Posts: 15623

PostPosted: Sat Jun 27, 2020 3:45 pm    Post subject: Reply with quote

extowgen wrote:
I can see there's an inherent problem in every OS, yet the problem is not due to a poor design choice, but rather due to human laziness.
I disagree with blaming laziness here. For the cases I cited, some were because no one thought the missed opportunity was worthwhile relative to other projects they considered important. For others, no one thought of the missed opportunity at all. They didn't even know there was an opportunity to miss. To me, laziness would suggest that they were aware of the opportunity, had the resources to make the change without seriously negatively impacting other objectives, and still decided not to.
Back to top
View user's profile Send private message
extowgen
n00b
n00b


Joined: 12 Jun 2020
Posts: 28
Location: online

PostPosted: Sun Jun 28, 2020 1:40 pm    Post subject: Reply with quote

Hu wrote:
extowgen wrote:
I can see there's an inherent problem in every OS, yet the problem is not due to a poor design choice, but rather due to human laziness.
I disagree with blaming laziness here. For the cases I cited, some were because no one thought the missed opportunity was worthwhile relative to other projects they considered important. For others, no one thought of the missed opportunity at all. They didn't even know there was an opportunity to miss. To me, laziness would suggest that they were aware of the opportunity, had the resources to make the change without seriously negatively impacting other objectives, and still decided not to.

The problem is not in something being missed.
Look at the situation in Windows: say your application needs B and C, B depends on D-1.0, C depends on D-2.0, and D has no version in the filename -- which means you're screwed.
There are situations where D actually has a version in the filename, but the version is always major. So now B and C depend on D-2, and D-2 has a minor update, but B was linked before the update and C after. On top of that your application can also depend on D-2. Which leaves you in a bad position, when there's no access to the source code of your dependencies.
I think, these two examples show the real problem.

Linux is absent of the problems just described, because it's not allowed to happen by having the ideology that everything has a maintainer, thus allowing having everything up to date, thus freeing filenames from versions, thus freeing development/build process from versions.
Yet, as I see it, Windows and Linux were faced with the same problem(lack of versioning, which I think is the result of laziness), but each produced a different solution, each solution having its downsides. Linux' solution being kind of solving it, Windows' solution is not doing anything and leaving it in the hands of developers.
Back to top
View user's profile Send private message
Display posts from previous:   
Reply to topic    Gentoo Forums Forum Index Portage & Programming All times are GMT
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum