[swift-dev] Swift incremental compile profiling

Samantha John sam at gethopscotch.com
Mon Apr 25 22:15:12 CDT 2016 

On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar <daniel_dunbar at apple.com>
wrote:

>
> On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev <
> swift-dev at swift.org> wrote:
>
> Hello List (cc/Jordan),
>
> At a high level: Brian and I are looking into contributing to incremental
> compilation in Swift. Right now we're trying to do an incremental compile
> by invoking swiftc.
>
> Our understanding so far:
> - swiftc takes a list of input files.
> - If swiftc is invoked with '-incremental', it requires an
> '-output-file-map' option to also be passed. We assume this is used to
> determine where intermediate dependency files go.
> - Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of
> files Xcode generates when building Swift projects, we determined
> '-output-file-map' to be a JSON file (which is parsed by the llvm yaml
> parser?
> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100).
> We tried writing our own, this is the shortest version that the compiler
> didn't error on:
>
> ```
> {
>   "": {
>     "swift-dependencies": "MyModule-main.swiftdeps"
>   }
> }
> ```
>
> - When compiling with the above file ('swiftc -incremental
> -output-file-map OurFileMap.json *.swift'), swiftc writes to
> MyModule-main.swiftdeps. The end result looks like this:
>
> ```
> version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift
> a2cf18ba2f)"
> options: "9277a78155e85019ce36a3c52e9f3f02"
> build_time: [514847765, 412105000]
> inputs:
>   "Class1.swift": [514841531, 0]
>   "Class2.swift": [514844635, 0]
>   "main.swift": [514841821, 0]
> ```
>
> - Where the [xxx, yyy] are timestamps with the first number representing
> seconds, the second nanoseconds. (
> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221
> )
>
> - We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift
> -parseable-output -save-temps' to show us the paths to the generated
> .swiftdeps files. We assume that to get incremental compiles to work for
> us, we'd need to pass these generated .swiftdeps files' paths to the
> compiler somehow. We can't figure out how to do this, so this is the point
> where our incremental compile fails:
> https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348
>
>
> You don't need to point the compiler at the swiftdeps files, it reuses the
> ones passed in the output file map if -incremental is passed.
>
> You didn't mention it (I don't think) but you also need to pass
> -emit-dependencies, however it sounds like you are doing this.
>
> This is as far as we got. Would super appreciate if anyone on the list
> could point us in the right direction from here, especially for the
> following questions:
>
> 1. This would probably be easier if we could find the right test case. We
> poked around in test/Driver/Dependencies which had a bunch of tests around
> reading the .swiftdeps files, but we couldn't find tests that demonstrated
> how incremental compilation worked at a high level.
>
> 2. '-output-file-map': Is this file meant to be written by hand, or is
> there a part of the swift compiler that writes this for you?
>
>
> It is meant to be written by the build system which invokes Swift. See
> also:
>
> https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206
> which is what the Swift package manager uses (which supports incremental
> compiles).
>

This sounds perfect. I tried to install the llbuild but got stuck on
filecheck (and saw that you are trying to add it to the toolchain:
https://github.com/apple/swift-llbuild/pull/22/files#r61007286) Is there an
executable of llbuild I could download, or do you have any basic
instructions for what to do about filecheck?


> Your best bet is to take the exact command line used by xcodebuild, and
> then invoke swiftc with that to replicate the incremental build. If you run
> with -v you should be able to see exactly what files get built
>

 I tried running xcodebuild with the verbose flag but wasn't getting much
useful. Is that what you meant here? -v didn't seem to be an option.

Thanks so much,
Sam


> HTH!
>  - Daniel
>
>
> 3. Specifying paths for .swiftdeps: We had assumed this was done based on
> the '-output-file-map', but writing the paths we wanted manually did not
> seem to work. Any tips?
>
> Thanks so much!
> Sam and Brian
>
> On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam at gethopscotch.com>
> wrote:
>
>> Hi Jordan,
>>
>> The thing that sticks out in the dependency analysis is the treatment of
>> external dependencies. The entire module has the same list of external
>> dependencies which causes a lot of needless recompiles- especially if you
>> start with a large swift project and slowly start to move things into
>> modules.
>>
>> So to me the lowest hanging fruit would be to only mark files for
>> recompilation that explicitly import the external dependency. This seems
>> pretty safe since you can't compile unless the dependency is explicitly
>> imported. Has anyone on the list tried this before?
>>
>> A second idea would be to consider a file as changed only if its build
>> artifact actually changes. Obviously, we'd have to actually build the file
>> to figure this out, so we wouldn't have the same level of parallelism
>> initially. Perhaps if it was an optional compiler flag this would be more
>> palatable? Also wondering if anyone has tried something along these lines.
>>
>> Thanks!
>>
>>
>> George- The bridge between objective c and swift between is definitely a
>> choke point. We've been able to mitigate objective-c recompiles somewhat by
>> limiting our imports of swift into objective-c. We've even gone so far as
>> to make wrapper classes in objective c around some of our most commonly
>> used swift classes so as not to import swift.
>>
>> It's also very true that changing an objective c .h file that is imported
>> into the bridging header will trigger massive recompiles. As more of our
>> app has transitioned to swift this has been less of an issue. Most of the
>> problems at this point have to do with recompiling a large portion of our
>> swift code due to small changes in unrelated parts of our other swift code.
>>
>>
>>
>>
>>
>> Get the latest from Hopscotch!
>>
>> Sign-up for our newsletter <http://eepurl.com/Ui0eX>
>>
>> On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists at gmail.com> wrote:
>>
>>> Hey Sam,
>>>
>>> One thought: if you have an app with mixed objc and swift code, then the
>>> app-bridge.h and app-swift.h files might be creating massive choke points
>>> in your dependency graph. I have no idea how optimized the bridging
>>> functionality is but it has always seemed like a potentially weak part of
>>> the dependency management. I imagine that with an objc half and a swift
>>> half of the code base, every time you make a change in a swift file you
>>> trigger recompilation of the objc half, and vice versa. I'd love to hear
>>> from the core team to what extent this is true!
>>>
>>> George
>>>
>>>
>>>
>>> On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <
>>> swift-dev at swift.org> wrote:
>>>
>>> Thank you Jordan! This is a great starting off point.
>>>
>>> I'm thinking about proposing a "strict import" mode in swift: A compile
>>> flag that when turned on would require you to explicitly import any file
>>> that contained a dependency you needed (like in objective-c).
>>>
>>> I'm going to spend more time looking over the docs and the output logs
>>> to see if this would be a feasible. If anyone has opinions or insights into
>>> this I would love to hear from you.
>>>
>>> Sam
>>>
>>> On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose at apple.com>
>>> wrote:
>>>
>>>> Hi, Sam. I don't think we currently have a good answer for this built
>>>> into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds
>>>> would be fast enough that it wouldn't matter! That's obviously not where we
>>>> are.)
>>>>
>>>> Since '-debug-time-function-bodies' is now public knowledge, I'll share
>>>> another one of our debugging flags, '-driver-show-incremental'. You can add
>>>> this to your "Other Swift Flags". The output isn't very detailed, though:
>>>>
>>>> Queuing Tree.swift (initial)
>>>> Queuing AdventureScene.swift (initial)
>>>> Queuing AdventureScene.swift because of dependencies discovered later
>>>> Queuing AppDelegate.swift because of dependencies discovered later
>>>> Queuing ChaseArtificialIntelligence.swift because of dependencies
>>>> discovered later
>>>> Queuing Character.swift because of dependencies discovered later
>>>> Queuing SpawnArtificialIntelligence.swift because of dependencies
>>>> discovered later
>>>> Queuing Goblin.swift because of dependencies discovered later
>>>> Queuing Cave.swift because of dependencies discovered later
>>>> Queuing AdventureSceneOSXEvents.swift because of dependencies
>>>> discovered later
>>>> Queuing HeroCharacter.swift because of dependencies discovered later
>>>> Queuing EnemyCharacter.swift because of dependencies discovered later
>>>> Queuing Boss.swift because of dependencies discovered later
>>>> Queuing SharedAssetManagement.swift because of dependencies discovered
>>>> later
>>>> Queuing Warrior.swift because of dependencies discovered later
>>>> Queuing Archer.swift because of dependencies discovered later
>>>> Queuing Player.swift because of dependencies discovered later
>>>> Queuing ArtificialIntelligence.swift because of dependencies discovered
>>>> later
>>>>
>>>> In this case, I took a version of the Adventure sample project and
>>>> modified "Tree.swift"; that triggered recompilation of several other files.
>>>> Unfortunately this view doesn't tell you how they're related, only which
>>>> ones are actually getting rebuilt.
>>>>
>>>> The next step (and moving into the territory of "working on Swift"
>>>> rather than just "trying to figure out why it's repeating work") would be
>>>> to look at the "swiftdeps" files stored in your DerivedData folder. These
>>>> are currently just YAML files describing what Swift thinks the file depends
>>>> on, as well as what will trigger rebuilding of other files. This is
>>>> intended to be a conservative estimate, since *not* recompiling
>>>> something would result in an invalid binary. (Unfortunately I say
>>>> "intended" because there are known bugs; fortunately, archive builds are
>>>> always clean builds anyway.)
>>>>
>>>> There's a document in the Swift repo describing the logic behind
>>>> Swift's dependency analysis:
>>>> https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst.
>>>> The one thing that's *not* in there is the notion of changes that
>>>> don't affect other files at all. This is accomplished by computing a hash
>>>> of all the tokens that *could* affect other files, and seeing if that
>>>> hash has changed.
>>>>
>>>> We definitely have room for improvement here.
>>>>
>>>> Jordan
>>>>
>>>>
>>>> On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <
>>>> swift-dev at swift.org> wrote:
>>>>
>>>> I have a large project (308 swift files, 441 objective c, 66k lines of
>>>> code) where incremental builds can be extremely slow. I'm trying to do some
>>>> profiling to figure out what type of things cause large scale recompiles.
>>>> The problem is that I can't find a good way of telling which files get
>>>> recompiled on an incremental build and which do not. It seems like files
>>>> that are not recompiled still get listed in xcode, but the compiler just
>>>> passes over them really fast.
>>>>
>>>> Does anyone know if xctool or xcodebuild has this type of
>>>> functionality? Or is there some other way to get this info?
>>>>
>>>> Thank you,
>>>> Sam
>>>> _______________________________________________
>>>> swift-dev mailing list
>>>> swift-dev at swift.org
>>>> https://lists.swift.org/mailman/listinfo/swift-dev
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> swift-dev mailing list
>>> swift-dev at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-dev
>>>
>>>
>>>
>>
> _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160425/b5e49943/attachment.html>

More information about the swift-dev mailing list