[swift-build-dev] Swift 4 Package Manager Roadmap

[Rick Ballard]

Hello Swift Package Manager community,

The package manager's release in Swift 3 was a big success, with many enthusiastic adopters. Since then we've been hard at work building the next version of the package manager. We have a lot we want to do, but focus is important for building a successful tool, so we've defined a set of goals that we expect to focus on for Swift 4. As always, this roadmap may be amended based on feedback from the community, and pull requests for improvements outside this list are still welcome!

This email covers a lot of different features, so instead of replying directly it would be best to start a new thread on any topic you'd like to discuss in detail.

Tomorrow (Tuesday, January 24th), from 10-11am PST, package manager developers will be available for a dedicated hour in the (unofficial) SwiftPM Slack channel to answer questions about this roadmap; that's a great time to drop in and chat. If you haven't joined the SwiftPM Slack before, you can get an invite to do so by entering your email at https://swift-package-manager.herokuapp.com <https://swift-package-manager.herokuapp.com/>.

Work We've Completed

We've implemented several large features and made many bugfixes and small improvements for this release already. The work we've already completed will also available in Swift 3.1; the remaining features to come will likely be in Swift 4 only. You can try out these changes now in trunk snapshots available on swift.org <http://swift.org/>. The most notable changes that we've made were:

– Package Edit

We've changed the workflow for making changes to a package's dependencies. Now, dependencies are stored in the tool-managed `.build` directory by default, but a new 'swift package edit' command allows users to "begin editing" on a package, moving it under the user's control (into the `Packages` directory), exempting it from dependency updates, and allowing the user to commit and push changes to that package.

For more details, please see the documentation: https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#editable-packages <https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#editable-packages>. The evolution proposal for this feature is at https://github.com/apple/swift-evolution/blob/master/proposals/0082-swiftpm-package-edit.md <https://github.com/apple/swift-evolution/blob/master/proposals/0082-swiftpm-package-edit.md>.

– Pinning

We've implemented the ability for the package manager to record the versions of each dependency you've used, and to "pin" or "unpin" specific packages to allow or prevent updating them to newer versions. This enables users to share the exact versions of dependencies they've used with their teammates or with their continuous integration system.

For more details, please see the documentation: https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#package-pinning <https://github.com/apple/swift-package-manager/blob/master/Documentation/Usage.md#package-pinning>. The evolution proposal for this feature is at https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md <https://github.com/apple/swift-evolution/blob/master/proposals/0145-package-manager-version-pinning.md>.

– Project Generation Improvements

We've rewritten our support for generating Xcode projects and fixed a number of bugs in this area. The new project generation support is backed by unit tests and is more maintainable and extensible than the original support.

– New Dependency Resolver

We've redesigned our algorithm for resolving the package dependency graph to fix correctness issues, and added significantly better infrastructure for working with source control. The package manager also now incrementally updates checked-out packages when performing `swift package update` instead of re-cloning dependencies each time. Better performance optimization of the dependency resolution algorithm remains as a to-do.

– Cycle Detection and Incremental Build Correctness

The package manager now detects dependency cycles, and has overall better diagnostics for this and other problems. We've also fixed correctness issues in our incremental build support.

– Parallel Cloning

We've added experimental support for cloning packages in parallel during dependency resolution, instead of cloning them one at a time. This feature is not yet on by default, but can be used with `swift build --enable-prefetching` or `swift package --enable-prefetching update`. This feature will be available in the snapshots on swift.org <http://swift.org/> soon.

– Improved Test Coverage and Infrastructure

We've improved the test coverage for the package manager project itself. We've also added significant new infrastructure, including a new command-line argument parser, low-level utilities, and deeper integration with Git.

Planned Work for Swift 4

These are the remaining features we plan to focus on for Swift 4:

– Swift Versioning Support

We need to define the package manager's behavior for handling the transition from Swift 3 to Swift 4. The Swift 4 compiler will support compiling source code using either the Swift 3 or Swift 4 language; Swift packages need some way to specify which language mode to use. In addition, we would like to be able to evolve the Package.swift manifest API, and to add new features which aren't supported by older versions of the package manager; we'll need a way to do so without causing packages that adopt these new features to break builds that are using an older version of Swift.

We are expecting to draft an evolution proposal for this soon.

– Product Types

We'd like to formalize the concept of "products" (such as libraries and executables) in Swift packages. This will allow better control over what types of products are produced (e.g. allowing users to request a static vs. a dynamic library), and will allow targets to depend on specific products produced by other packages instead of needing a broad dependency on an entire package.

Implementation of this feature is in progress. For more details, please see the evolution proposal: https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md <https://github.com/apple/swift-evolution/blob/master/proposals/0146-package-manager-product-definitions.md>

– Support for Top-of-Tree and Branch-Based Package Development

Currently a Swift package can only depend on tagged releases of its dependencies. This makes it difficult to use the package manager to work on top-of-tree of a package graph, or on other branches. This is a workflow that we'd like to support.

There is a draft evolution proposal in progress at https://github.com/neonichu/swift-evolution/pull/1/files <https://github.com/neonichu/swift-evolution/pull/1>

– Overriding Package Conventions

The package manager provides certain conventions for the on-disk layout of a package which allow us to identify multiple modules, infer product types (executable vs library), associate tests with the module they test, etc. If you wish to create a Swift package from a source tree which does not conform to these conventions, there is currently no way to do so. We'd like for users to be able to override the conventions and explicitly specify how their source tree is laid out using directives in the Package.swift manifest.

We are also considering removing some of the existing conventions so that there is one standardized conventional layout instead of the multiple supported layouts available today; if we do so, manifest convention overrides would allow existing packages that are using the alternate layouts to continue using those layouts.

– Support Build Settings

The package manager has no way to provide specific settings, such as compiler flags, for building the targets in a package. Some settings are critical for production code, such as the "deployment target" (the minimum OS version that the built code will run on). We need to provide a way to set at least the most important settings which users will need control over.

– Redesign the Package Manifest API

Our Package.swift manifest API was designed prior to the Swift 3 language conventions, and its design was not reviewed by the evolution process. We would like to redesign this API as necessary to provide a clean, conventions-compliant API that we can rely on in the future.

– Performance Test Support

Currently, the package manager has no support for running performance tests (as distinct from correctness tests).

There is a draft evolution proposal in progress at https://github.com/aciidb0mb3r/swift-evolution/blob/pref-proposal/proposals/xxxx-package-manager-performance-testing.md <https://github.com/aciidb0mb3r/swift-evolution/blob/pref-proposal/proposals/xxxx-package-manager-performance-testing.md>

– SwiftPM-as-a-Library

The package manager should expose a library API for all of its commands, which will allow it to be integrated into an IDE. Initial work on this API is in progress, though the current API in development should not be considered a stable public API. Any clients adopting the API in progress should consider it subject to change in any release of Swift.

– Configuration File Support

We are considering adding a configuration file mechanism, which could be user-specific or package-specific, to allow customizing certain package manager behaviors. One immediate use for this would be as a temporary mechanism for passing overriding compiler flags for a package, though we hope to solve that need with a real build settings model. Other uses for a configuration file are up for consideration.

– Multi-Package Repository Support

Currently the package manager requires that each package live at the root of a Git repository. This means that you can't store multiple packages in the same repository, or develop packages which locate each other in a filesystem-relative manner without relying on Git. We need a proposal for how we would like to support this.

There is a draft evolution proposal in progress at https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md <https://github.com/ddunbar/swift-evolution/blob/multi-package-repos/proposals/NNNN-swiftpm-multi-package-repos.md>

Beyond Swift 4

Swift 4 will be a big step forward for the package manager, but there are many features we don't have time for in this release. Upcoming features we'll be considering after Swift 4 include a centralized package index, support for package resources (such as image assets), license and metadata support, explicit support for handling source control forking, and a generic mechanism for invoking build tools that the package manager doesn't natively support.

- - -

We really appreciate the enthusiastic adoption and feedback we've gotten from the package manager community. We'll be bringing each of these topics to discuss with the community as we tackle them, so look for separate mailing list threads about each feature in the future.

Thank you all for your support!

	- Rick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-build-dev/attachments/20170123/38a82919/attachment.html>