[swift-build-dev] Hello SPM

Mon Dec 7 11:30:01 CST 2015

> On Dec 7, 2015, at 5:22 AM, Eloy Durán via swift-build-dev <swift-build-dev at swift.org> wrote:
> 
>> Does CocoaPods maintain a roadmap for what features are planned / scheduled?
> 
> 
> The scheduled ones at the moment are in the 1.0.0 milestone https://github.com/CocoaPods/CocoaPods/milestones/1.0.0

Cool, thanks.

>> Absolutely, we would welcome the same. Was there a specific discussion you were hoping to start on this thread?
> 
> 
> For now just the exact one that we’re now having, you have some great questions already.
> 
>> Thanks for reaching out, I would love to find ways for all of us to collaborate. I think we all want the same thing -- beautiful, easy, scalable development and a rich ecosystem.
> 
> 
> Definitely.
> 
>> Here are some of my random initial thoughts/questions/ideas from a purely technical perspective:
>> 
>> 1. Is CocoaPods technically able to use Swift-based libraries? If there were clear factorings of support functionality that we could build as, say, SPM packages, would CocoaPods be able to use them?
> 
> Yes, it does support them right now. Here’s an example lib of one of our core team members (Kyle Fuller): https://github.com/kylef/stencil and its specification https://github.com/kylef/Stencil/blob/master/Stencil.podspec.json.

Sorry, I wasn't clear here.

What I meant was is the *implementation* of CocoaPods actually able to use Swift libraries. I'm wondering if it would ever be viable to factor out shared libraries that both projects used, but I can imagine that using (and installing) those libraries from the Ruby based CocoaPods might be non-trivial?

> One of our other core team members (Boris Bügling) has already started on a Swift tool to generate CocoaPods specifications from SPM specifications: https://github.com/neonichu/Chocolat. Some metadata that we deem important, at least in our centralised service, is missing from SPM specifications: https://github.com/neonichu/Chocolat/issues/2. One of those is a license, which you already mention in your docs as something you’d be interested in looking into, the other is a description of the library, which you might also need if/when you’d ever build an index, adding these sooner rather than later would make it much easier for Boris’ tool to work perfect.

I think both of those are things it makes sense to add soon (and encourage people to author).

> I’ll let Boris do the speaking about what we’d really need here and to answer further questions on this front.
> 
>> 2. The chicken and egg problem is something which makes evaluating design choices very difficult. Informally, the rule that Max and I have often talked about is that "80%+ of projects" should be able to use the convention based system with only the "leading package specification" (let package = Package(...)) and no other custom configuration. However, we can't currently objectively evaluate whether a particular design choice satisfies that criteria without a large body of projects that already work (or almost work) with SPM. I would like to investigate whether we can make use of CocoaPods existing corpus of projects (and infrastructure for testing them) for these kinds of purposes. This would help us quantify things like "this is how much work X percent of projects have to do to adopt SPM" and  "X percent of projects need feature Y to work”.
> 
> 
> We have quite some data on the thousands of libraries that we currently support and even stats on integration which might be useful to you. I’ll leave it to yet another of our team members (Orta Therox) to give you access to that data and show you around.

Ok, great! Do you have existing infrastructure for analysis of those libraries? Would it be easy to write a tool to, e.g., count #s and types of libraries? Or create a summary of source layout styles?

>> 3. One of my personal beliefs is that part of the reason software packaging & integration has historically been so messy is because no one has been able to be a "force for change" and cause projects to adapt. For C/C++/Obj-C projects, there are a lot of "unnecessary" variations in the project structure which add complexity for very little overall gain. For example, some things are simpler if you can assume that any package X installs its headers neatly under an "X" subdirectory -- but since historically this was never done, no has ever been able to encourage projects that don't do this to change.
>> 
>> One of my hopes is that, eventually, Swift and SPM support can function as this force for change. If we can come up with a stricter set of requirements that improves our ability to make a very reliable, smooth development experience, then it may be that the value of supporting Swift and SPM is enough to encourage upstream projects to undertake the pain of changing their code base to meet those requirements.
> 
> I think that it will be quite easy for you to go forward with the strict structure approach and you should totally do so. Swift is still new, the people are excited to work on their projects and are already used to having to update them as Swift matured, and last but not least, you are Apple :) Many of the people that did not want to use CocoaPods would not do so because it’s not how Apple did things and some of those were definitely technically valid as we couldn’t integrate as nicely with Xcode as Apple ever could.
> 
> Having said that, I think this really only applies to Swift libraries. Existing Objective-C libraries might be doable if the author is using their lib in Swift or otherwise interested in overhauling their project’s layout, but C/C++ projects will prove hard unless Swift users are going to want to go the extra mile to overhaul project layouts for those, but these can get very hard to get right. Right now you’re making the choice to tell people to install those differently (e.g. homebrew), but that makes integration a bit harder for users, especially for a platform like iOS. And that’s not even considering the issue of how to link those dependencies into an app.

Right. We do want to support C-family projects natively (not just system dependencies) in the long term. As you know much of the Swift project itself is in C++; we are pretty highly motivated to support this just to have a better development / build model for ourselves.

I hope to eventually be able to apply leverage on C-family projects, too. There is still no good way of Integrating C-family projects into one another; anyone who has tried to write a small tool that uses LLVM will know exactly what I am talking about. In some ways, this situation is actually even worse than for Swift projects because of the diversity and complexity of C-family project build systems. I believe that if we could support some strong conventions for C/C++ projects, to the point that it was suddenly easy to build dependencies together, then we would have a very compelling system that project owners would actually be excited to support.

One evolutionary approach I have considered is that we could, for C-family projects, provide built in tools to define adaptor packages which integrate non-conforming projects into the system. This is analogous to how we support module-map packages today, even though in an ideal world projects would ship module definitions with their project. This would allow the ecosystem to build up, and people would begin to see the value of it. At that point, the owners of the adaptor packages could upstream their work into the project itself. And then we would all live in a better world! :)

>> The reason I bring this up in relation to CocoaPods is that I suspect CocoaPods is currently in the camp of "we have a large body of existing clients and cannot aggressively try and simplify our model if it means they need to change", whereas for SPM we are much more likely to be in the camp of "we would like to require all projects to behave this way, even if it means many existing projects will need to be adapted, because we think it will allow us to build a better experience in the long run".
>> 
>> What is your perspective on this problem? How big of an issue has this been for CocoaPods to evolve?
> 
> You’re absolutely right about these problems, they have been very real. When we started on CocoaPods, we were entering into a long established ecosystem and we too had a chicken and egg problem. There was no way that people would adopt their existing projects to CocoaPods, so it had to basically work with all of them. I absolutely love convention over configuration, but this definitely made that very hard to do so. The hardest part has always been projects that use Xcode projects:
> 
> * Because these documents are too opaque to properly reflect upon and e.g. ensure a project doesn’t vendor a third party lib that could lead to clashes or what other libs and the versions thereof the project actually depends on. This was especially hard with libraries where the one maintaining the CocoaPods specification wasn’t the lib’s author and wasn’t always inclined to do their absolute best to ensure the specifications validity for everybody. However, over time people did start going CocoaPods first with their libraries, which made this problem largely go away and was a big win in transparency of what dependencies you were using imo.

Yup. One of our very explicit goals is to minimize the the amount of stuff that is "opaque" to swiftpm, so that we at least have an open road to very aggressively auditing what a package does and assisting users in writing the definition correctly. There is a tension between this goal and quickly supporting all of the things people want to do (especially w.r.t. C-family projects), but I think it is important that we get this right.

> * Integrating the libs into the user’s project in a way that made the most sense from the way people were doing development with Xcode, which are including the source so that they are always fully navigable and debuggable, meant we had to generate Xcode projects ourselves. We have never been happy with this, because the undocumented/unsupported nature of it tends to lead to a lot of friction. I’m hoping that the way SPM is going to integrate into Xcode might also open up a possibility for CocoaPods to integrate artefacts without having to deal with Xcode details or possibly by using an Xcode specific simple, documented, and supported specification on how to integrate artifacts.

I understand the tension of needing to generate projects and the downsides. I have a lot of experience working with LLVM's auto generated (via CMake) Xcode project, and it is just very difficult to build a seamless experience when adapting two very different project models this way.

We are trying something new here, which is to build a package manager that exposes an API, and then rely on clients to build good integration by using that API well. TBH this is a bit of an experiment, but I believe based on our experience with libclang/SourceKit that it is absolutely the right way to go and will let us do something great. Of course, this is all currently TBD. We don't yet vend a formal library much less any real APIs, so this plan will become more fully formed in the coming months...

> I hope I addressed all your questions to the full extent, please do let me know otherwise.

One of the specific things I was curious about was how CocoaPods has historically handled changes to the podspec definition format? How does this work if a podspec and its dependencies start requiring incompatible versions of CocoaPods itself? Have you just managed  to avoid this, or are there legacy features you have ended up supporting but wish you didn't need to? I see things like issue #840 which I assume reflects some of how this worked out in practice.

 - Daniel

> 
> – Eloy
> 
> _______________________________________________
> swift-build-dev mailing list
> swift-build-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-build-dev