My beef with composer and Magento

magento-composer-issues.md

My beef with composer and Magento

Magento has recently merged an architecture proposal with the goal of removing non-composer modules.
At first this seems like a good idea, but I think there are some problems.

Community Bias

Magento listens to the community. That is a good thing.

However, the loudest voices in the developer ecosystem are extension vendors. Both Fooman and Yireo have blogged about why module development in app/code is "wrong" or "bad".

I love Christof and Jisse - this is not a personal attack, they are doing outstanding work. But they fall pray to our usual human bias.
They are heavily involved in their extension businesses, while I've mainly worked as a freelancer in merchant developer teams over the last three years.
They fail to see the perspecive of non-extension vendor developers.

There are no evangelizing blog posts by merchant developers that I know of. In general, merchant developers seem a lot quieter and not as involved in the community.
The set of constraints a merchant developer is working under is very different from the set of constraints of an extension vendor or an SI developer.
I feel that Magento is missing that perspective.

A merchant developer

Composer is a natural fit for an extension vendor, and much has been written on the benefits of using composer to manage modules. I'll focus on the merchant developer point of view in this post.

For a merchant developer, using composer to manage modules dependencies has zero benefits and is costly.
Let's have a look why I say such a preposterous thing.

The big difference is that for a merchant developer, custom modules will only ever be installed in one Magento instance.

The constraints for a developer working for a merchant are:

• Modules are developed together, not stand-alone
• Modules will only ever be installed on one Magento instance
• Modules will be upgraded together with Magento

These differences negate all the benefits of working with custom modules as individual packages installed by composer.

So, what are the benefits of listing composer dependencies on individual modules, and why don't they apply to development in a merchant team context?

• Installing dependencies, i.e. finding compatible Magento module versions.
  The dependency will always be the current Magento instance. The modules will never be installed on different Magento versions. Dependencies on custom modules are already resolved because they are right there, in app/code, too.
• Module conflicts don't happen, as all merchant modules are developed together.

What about agency or SI developers?

For agency developers it depends on their business model and client structure. It can make sense to follow an approach where modules are developed individually, or it can make sense from a cost and development perspective to keep all custom modules together in app/code.

The costs of composer modules

Using composer to manage module dependencies is costly, firstly in time, which as we all know translates directly to money:

• Tracking dependencies on Magento.
  Every time I write code that uses a part of the Magento platform I have to stop and think if I already have the appropriate dependency in the module composer.json.
  And even if that module or framework dependency is already listed, I have to ask myself if the code I introduced causes a change in the dependency version constraint.
  This breaks my flow and forces me to spend time checking the module service contracts and Magento rules for module version dependencies.

• Keeping dependencies between custom merchant modules in sync.
  When I change a module, I have to make a release so composer will see the update. That means I have to keep track of the type of changes I make and how other custom module use my module, and upgrade those dependencies, accordingly.

• More moving parts, more can go wrong.
  Having to track all those additional dependencies increases the risk of getting into dependency hell (see below).

• Build time
  The more packages in the dependency graph, the longer composer takes to create a build. This might not be a lot for individual builds, but it sure adds up over time.

• Creating new modules takes longer and is more complex.
  Instead of just running a scaffolding tool or creating the files manually in app/code and be done, I have to create a stub module composer.json, add a path repository configuration to the root composer.json, install the module with composer, exclude the new folder under vendor from being tracked in PHPStorm, and then I can finally start writing some code. Again, this isn't too bad once, but it does add up over time a lot.

Magento's module dependency rules

I have yet to meet a developer who knows all of Magento's module dependency rules. In addition to those rules, there also are the rules how versions must change according to code base changes. And finally there is also the modularity section in the technical guidelines.

If you haven't read these pages, go, read them now. Then continue to read this post.

The rules are too complicated to keep in mind while focused on solving a business problem.

During trainings, I've found it is hard enough for developers to wrap their head around when to use require and when to use suggest.
The sad thing is, if I follow those rules and specify all individual module dependencies accordingly, I can't even run unit tests in CI after a composer install, let alone integration tests...

The rules are counter-intuitive and hard to apply. Most of the time my modules end up with mostly patch level dependencies pretty quickly.

Most of the outspoken people in the PHP developer ecosystem argue that a deep understanding of composer and semver are a basic requirement nowadays. But that isn't my experience. Most PHP developers I meet aren't that deeply familiar with composer. Many have to google basic comands every time they use them.

Forcing Magento developers to memorize and apply all those rules, or maybe even only a subset of those rules, makes it even harder to find PHP developers that already "know" Magento.

Arguing that composer is a standard and as such it's easy to find talent is simply not true, if pages and pages of rules have to be specified in addition to "we use composer".

Upgrading Magento

Let's think about the steps to upgrade Magento.

Regarding composer, we run a variation of this, where the exact version and meta package might differ:

composer require magento/product-community-edition=2.3.1 --no-update
composer update
bin/magento setup:upgrade

This will make composer find all the versions of all installed packages that are compatible with the meta-package and install them.
And if there is no compatible module, the upgrade will be aborted.
This is great from an extension developers point of view.

But from merchant developers point of view, this is the start of a world of pain if module dependencies are tracked with composer.

In order to do a test upgrade, I have to run the above command and analyze the output of composer to figure out which of my modules have incompatible version constraints. Then I have to go into each one, change the constraints so Magento can be upgraded.
I have to do that for all custom modules at once.
And then I can check if the version conflict actually broke a custom module or if the change doesn't really affect the custom module.

Just going through the composer output and changing module dependencies so Magento can be upgraded takes hours on a large instance.

If a version constraint prohibits Magento from being updated, that should indicate the package in question is not compatible.
However, most of the time, that module actually still is compatible.
Code changes in a package that cause an incompatible version bump don't mean the package actually is incompatible!
This is the major downfall of semver.

What upgrading Magento boils down to, is that we always have to test if custom modules are compatible with an upgrade.
It simply is impossible to express runtime code dependencies in semver.

As a merchant developer, there is no net benefit in composer aborting an upgrade if a version constraint in a custom module is incompatible.
The same amount of testing is required to see if the new version of Magento really is compatible or not, with or without that dependency declaration.

The base problem is that from a merchant developer perspective, all custom modules implicitly depend on the meta package version, and not individual packages.
This is an implicit dependency, since we are talking about Magento modules. They will only ever be used in the context of Magento.
So there is no benefit in specifying even the meta package version dependency in custom modules.

Without composer dependencies listed in custom modules, doing a test upgrade is a lot simpler (and much much faster).
Simply running the composer command to install the new Magento version enables a merchant developer to check for broken modules, without having to spend hours updating version constraints to get to that point.

External non-Magento deps

Another argument that is often used against keeping modules in app/code is that of non-Magento dependencies. When a custom module depends on a non-Magento package - for example something from packagist - composer won't see that dependency in the modules composer.json file and it won't be installed.

Luckily this isn't a problem thanks to the great composer merge plugin.
Nuff said, let's move on.

Dependency hell

Since I've used this term, I would like to briefly specify what I mean.
In terms of composer, dependency hell is when I want to install or update a package, but composer refuses to do that because of conflicting version constraints.
Usually it is triggered by two or more packages requiring incompatible versions of another package.
If these are direct dependencies, it's bad enough, but if they are version conflicts in transitive dependencies, it gets really ugly.
This can lead to hours of yak shaving, where fixing one version constraint only leads to the next conflict.

In summary...

... it's valuable to be able to use non-composer packages in Magento.

Anton Krill argues:

My understanding is that if we solve the performance problem, it will reduce complexity:

• less framework code to maintain
• less things for a new Magento developer to learn
• less boilerplate code

I disagree. It is easier to use module.xml and registration.php than to apply Magento's module dependency rules and semver.
For an extension vendor and some SIs it may be worth the cost, but for merchant developer teams it most certainly isn't.

Taking away non-composer modules will increase development cost for merchants and make it harder to hire Magento developers.

Thanks for the detailed post, Vinai. It describes real problems we rarely hear about.

I don't think the proposal you referenced argues with any point you brought up as it does not:

• ban app/code. I agree that project code should be developed in project folder, not separate composer packages.
• force you to specify SemVer dependencies in custom code. I'm not a big fan of SemVer.

The proposal is about minor file duplication removal. Banning app/code would definitely require a more detailed and motivated proposal. I think the part that causes confusion is the point about relying on Composer for dependency management. It needs clarification.

You can avoid most of the dependency resolution problems you described by specifying only major versions or stars in dependencies if you don't need the notification feature of Composer. But I agree that SemVer is not a good notification mechanism.

Your perspective is valid, and the problem is big, however in some points I dont agree with strong statements like "Keeping dependencies between custom merchant modules in sync. When I change a module, I have to make a release so composer will see the update.".
Many pain points you have mentioned can be resolved (just best practices would need to be documented).
For example:

1. Using composer you can still have one repository with your project and custom project specific modules. Composer supports config like

"repositories": [
    {
      "type": "path",
      "url": "./modules/*",
        "options": {
          "symlink": true
      }
    }
  ],

2. So you can still have all the project modules developed together. You don't have to publish modules somewhere just to download them using composer.
3. You don't have to raise versions to get new version of the module being used.

If you want to do a little brainstorming together, just let me know.

@tmotyl Thanks for picking up on the point of a local composer path. Indeed, my original point was to use a local path for project-specific modules and use the wildcard, so that the repository links are only added once. To my opinion (and experience), this adds little additional overhead. It was actually the main "trick" I tried to bring up in my blog, but it seems nobody was picking up on that yet. To further expand on this, there is no tagging within this local composer path either, so the version of each composer module served via this local path is always @dev (which basically equals it to the project itself).

Thank you for writing this up, @Vinai. I can't say I fully understand the proposal as I'm still neck deep in Magento 1, but as a merchant developer I trust that I will share your viewpoint once we start the migration to Magento 2.

Thanks for the replies!

@antonkril
Let's keep the discussion in the PR threads. I responed to the your new PR there.

@tmotyl
That's a neat trick, I wasn't aware that paths repository urls could use glob syntax!
That removes the problem of having to add one repository configuration to the root composer.json for each module, but not more.
However, using local composer path repositories has some problems.

A common pain point we experienced where git merge conflicts for the composer.lock file (for merges or rebases).
These are tricky to resolve manually since the file is generated. Usually the solution is to delete the conflicted composer.lock and have composer rebuild it (hoping it workes with a half-merged project), and then committing the new one. Alternatively any of the two file versions can be used to resolve the conflict and then rebuild the file after the merge is complete. But in any case, rebuilding the file takes a long time and any git conflict is a stressfull developer experience. This alone makes the whole local composer repo idea pretty ugly to me.
Another issue I've experienced when working on shops using local composer repositories for project modules is that when versions are specified, composer complains about missmatches to the composer.lock file when those versions are changed. I realize versions are optional in this case, but it happens, especially if developers copy&paste a composer.json from core modules as a base to get started.
Again, composer is not a tool all PHP developes know deeply.
So all in all my experience with using local repos for project code is not very nice.

@jissereitsma
I get that you wanted to promote using local composer repos, but I don't think they are a good alternative to development in app/code for project modules (see above).

@Vinai @tmotyl It is nice to discuss some practices on the local composer repos bit. Indeed a composer.lock git conflict could still occur and referencing more modules in the lock file increases this risk. However, I would only recommend using the local composer repos with version @dev (and not specific versions) because a local path doesn't have git tags or multiple version paths or something. Because of this, you never need to run composer update foobar/my-local-path-module. And with this approach, I've not heard of any git conflicts.

@Vinai The point on many project modules needing to over-maintained when using composer, landed with me. I'm convinced in that case, the app/code approach is much easier.

@jissereitsma
Happy I managed to reach you :)
The merge conflicts happen when feature branches get merged. Each branch can contain one or more new modules that where installed with composer.
In that case the composer.lock will differ in each branch and the conflict will occur.