[swift-dev] Swift incremental compile profiling

[Jordan Rose]

[+swift-dev again, for posterity]

Let’s see, those two are really mostly “I haven’t thought about how to recover from this scenario.” I have an internal presentation on why dependency analysis is the way it is that I should probably throw up in the docs in some format, but mostly these are all the occasions where dependencies change during a build, and I haven’t convinced myself one way or another whether the swiftdeps file left over from the last compilation is sufficient to represent what happened to a file that’s been removed.

Tasks more directly related to incremental builds aren’t really represented in JIRA because there aren’t many in digestible chunks. The most obvious improvements are all wrapped up in other things:

- Moving operators into types has the potential to stop making “==“ a top-level dependency on everything that defines “==“, but we haven’t figured out how yet. (Being part of a constraint system instead of being looked up as a member makes that hard, even though if we pick a member operator it must operate on one of the types involved.)

- Vtable layout depends on the methods in a class, so even if file A only calls one method in a class, adding a completely different method forces file A to be recompiled. This is probably the right trade-off in release builds, but not debug builds.

- Dependencies only use base names rather than full names, but default arguments make it hard to specify which full names are affected when a particular method is changed.

There are also two Radars about places where the incremental build logic seems to be incorrect; unfortunately I can’t share those projects. Obviously a bad incremental experience is still preferable to behavior changing on a clean build.

Tasks related to incremental builds also aren’t represented so much in Driver because at this point a lot of it’s about breaking dependencies and not being overconservative. The Driver generally can’t help with that; it just deals with the dependencies that the frontend spits out. The one thing we could do here is improve -driver-show-incremental. I haven’t used it in a while so I don’t actually know where I left off with that.

One thing we have noticed is that the cost for each frontend task for a project with 1500 tiny, non-dependent Swift files and a bridging header is much higher than it is for Clang. People have blamed Swift having to open and parse every file in the project, but I don’t actually believe this is the case; time spent in Parse is tiny for these functions. It would be very interesting for someone to investigate where this time is going. (There’s a similar project in SR-2461 <https://bugs.swift.org/browse/SR-2461>; I haven’t checked whether the project we have in Radar is appropriately clean of user info to share.)

I guess that’s about it. Oh, and I keep meaning to write tools to manipulate the data in the swiftdeps YAML files, but that can happen at any time. You’re welcome to contribute something before I get to it.

Definitely thanks for helping out here!
Jordan


> On Aug 26, 2016, at 11:35, Brian Gesiak <modocache at gmail.com> wrote:
> 
> Although I was hoping to do more work on this over the weekend, so I'd really appreciate any pointers you had before then.
> 
> I'm currently looking into the "file removed" case that causes a complete rebuild -- the "FIXME: Distinguish errors from "file removed", which is benign." and "FIXME: Can we do better?" comments in lib/Driver/Driver.cpp. So if you have any insights into that, or really anything else, I'd love to hear them!
> 
> - Brian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160826/bc59a1e6/attachment.html>