[swift-evolution] [swift-build-dev] [Review] SE-0145: Package Manager Version Pinning (Revised)

[Daniel Dunbar]

Hi Alex,

> On Nov 24, 2016, at 4:54 AM, Alex Blewitt via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
> 
>> On 20 Nov 2016, at 05:48, Anders Bertelrud via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>> 
>> 	* What is your evaluation of the proposal?
> 
> A way of building against a specific set of dependencies, rather than a variable set (or master), is needed in Swift. The current build process seems opinionated (but not mandated) to check these explicit dependencies into version control. If dealing with builds-from-source (as currently stands in Swift) then building from a moving target makes sense, and is likely the only sane default, but when binary dependencies are generated the moving target makes less sense.
> 
> At the moment, the proposal suggests having a secondary 'pins' file, which exists to allow explicit dependencies to be checked in to version control. This can be done at the moment, using a Version(1,2,3) or range Version(1,2,3)...Version(1.2.3) in the constraints.
> 
> When dealing with unreleased software, there is no explicit version - and thus the resolution can't apply. In effect, a version control hash takes place of an explicit version for the purposes of the dependency resolution. Being able to override the version resolved with a hash, without making changes to a checked-in file, seems like a good idea.
> 
> What the proposal doesn't bring forward clearly is the versioning of transitive dependencies, and whether or not those dependencies violate the requirement constraints. For example, if A depends on B, and B depends on C version 1.2.3+, can or should A pin C version 0.9.9? What if A depends on B and D, both of which depend on different versions of C that may be semantically equivalent (e.g. C version 1.2.3 and C version 1.4.5)? This will come up more often than the 'fail to build' approach outlined in the proposal.
> 
> It would be useful if the tool to update a version dependency could allow for dependencies which used to be present but which have been since removed could be cleaned up, optionally with a warning.
> 
> For pin files that are version controlled, having each pinned dependency on a separate line (at least) will permit sensible version control introspection. Any start-of-list parenthesis or commas should be designed in such a way that removing the first pinned dependency or last pinned dependency results in an SCM diff which just has a single line change (so, for example, trailing commas would be useful in any list structures).
> 
> Another useful mode would be to re-write the version ranges to increase the lower limit of the dependency in the Package.swift file. If code has been built and tested against a library version 1.2.3, then it may not be appropriate for that build to be built against anything lower. Increasing the lower bound to match (or having a command to do the same) would be a way of keeping track of the moving dependencies for the package itself.
> 
> The proposal also doesn't make clear to me what the dependencies are named for. The expectation is that developers will name the target for the repository name that is referring to. Should that short name be used as a dependency, or the URL? What if the target's URL has been replaced with another (say, because you're working on a local fork of an existing project's component?)

I'm having difficulty understanding here what you are concretely proposing. But here are my answers to the questions I think you are asking (or changes you are requesting):

1. The reason we want the manifest to have the range, and the pin to have the pin, is that we want the manifest to specify the "ideal range" w.r.t. semantic versioning. So it is incorrect to over-specify the range in the manifest by pinning, that breaks the use of semantic versioning in other packages. Hence, we store the pin data in a separate location and have a separate workflow for managing that data (whether that data is checked in is left to the project).

2. I agree we will need mechanisms to pin to a SHA.

If this doesn't answer your questions/concerns, can you restate them individually if this was a request for changing the proposal (something unlikely to happen at this point, but we can always amend the behavior subsequent to implementation)?

 - Daniel

> 
>> 	* Is the problem being addressed significant enough to warrant a change to Swift?
> 
> Yes. At the moment, it is not possible to use swift build to rebuild an old version of a product, without manually cloning many repositories and adjusting their hashes to match. Having the tool handle these dependencies automatically will manage this for the developer, and allow a step back in time to find out what the dependencies were.
> 
>> 	* Does this proposal fit well with the feel and direction of Swift?
> 
> Yes, this proposal is a natural extension to the way in which swift build works and will ensure repeatable builds.
> 
>> 	* If you have used other languages or libraries with a similar feature, how do you feel that this proposal compares to those?
> 
> Many languages who have such a dependency management mechanism do so by putting the versions next to the dependency in the format, and the dependencies are updated in the top level when they are needed. For example:
> 
> Maven: https://maven.apache.org/pom.html#Dependencies <https://maven.apache.org/pom.html#Dependencies> 
> Gradle: https://docs.gradle.org/current/userguide/artifact_dependencies_tutorial.html#sec:declaring_your_dependencies <https://docs.gradle.org/current/userguide/artifact_dependencies_tutorial.html#sec:declaring_your_dependencies> 
> Ivy: https://ant.apache.org/ivy/history/latest-milestone/ivyfile/dependency.html <https://ant.apache.org/ivy/history/latest-milestone/ivyfile/dependency.html> 
> 
> Although these tools allow specifying ranges of versions, this has been effectively deprecated over time as they don't provide for repeatable builds. Instead, the exact version (in the form of major/minor/micro) is encoded in the dependency itself, which allows the build to be repeated subsequently. (Dependencies in this world are identified with a fixed/immutable version number.)
> 
> One advantage of having the versions in the main dependency file is that it allows a view over what the history of those dependencies are simply by using the version control log. There are also other tools that can update the versions to the latest (or latest major, latest minor etc.) which update the dependencies in place.
> 
> http://www.mojohaus.org/versions-maven-plugin/ <http://www.mojohaus.org/versions-maven-plugin/> 
> 
> What these tools don't have is the ability to record a non-released version of a dependency; for example, if you know a but is fixed in a commit abc123 there isn't a way of depending on that.
> 
> Finally, one maven encodes the dependency information in the generated artefact. This allows the JAR to be unzipped and the dependencies investigated. For example, the dependency for the slf4j artefact (in the POM) is stored within the JAR in the META-INF/maven/org.slf4j/slf4j-log4j12/pom.xml of the generated code, along with a pinned stamp of the dependency used in the pom.properties file:
> 
> http://search.maven.org/remotecontent?filepath=org/slf4j/slf4j-log4j12/1.7.21/slf4j-log4j12-1.7.21.pom <http://search.maven.org/remotecontent?filepath=org/slf4j/slf4j-log4j12/1.7.21/slf4j-log4j12-1.7.21.pom>
> http://search.maven.org/remotecontent?filepath=org/slf4j/slf4j-log4j12/1.7.21/slf4j-log4j12-1.7.21.jar <http://search.maven.org/remotecontent?filepath=org/slf4j/slf4j-log4j12/1.7.21/slf4j-log4j12-1.7.21.jar>
> 
> The pinning here takes the form of a major/minor/micro version, rather than a commit hash, but is similar in effect. (Unlike the constraints in Swift's package manager, a version in Maven dependencies is an exact version, not a range).
> 
>> 	* How much effort did you put into your review? A glance, a quick reading, or an in-depth study?
> 
> I spent some time going through the proposal and am familiar with other dependency management systems which have gone through a different evolution.
> 
> Alex
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20161202/d9607101/attachment.html>