[swift-dev] Swift incremental compile profiling

Samantha John sam at gethopscotch.com
Sun Apr 24 17:19:13 CDT 2016 

Hello List (cc/Jordan),

At a high level: Brian and I are looking into contributing to incremental
compilation in Swift. Right now we're trying to do an incremental compile
by invoking swiftc.

Our understanding so far:
- swiftc takes a list of input files.
- If swiftc is invoked with '-incremental', it requires an
'-output-file-map' option to also be passed. We assume this is used to
determine where intermediate dependency files go.
- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of
files Xcode generates when building Swift projects, we determined
'-output-file-map' to be a JSON file (which is parsed by the llvm yaml
parser?
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100).
We tried writing our own, this is the shortest version that the compiler
didn't error on:

```
{
  "": {
    "swift-dependencies": "MyModule-main.swiftdeps"
  }
}
```

- When compiling with the above file ('swiftc -incremental -output-file-map
OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The
end result looks like this:

```
version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift
a2cf18ba2f)"
options: "9277a78155e85019ce36a3c52e9f3f02"
build_time: [514847765, 412105000]
inputs:
  "Class1.swift": [514841531, 0]
  "Class2.swift": [514844635, 0]
  "main.swift": [514841821, 0]
```

- Where the [xxx, yyy] are timestamps with the first number representing
seconds, the second nanoseconds. (
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221
)

- We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift
-parseable-output -save-temps' to show us the paths to the generated
.swiftdeps files. We assume that to get incremental compiles to work for
us, we'd need to pass these generated .swiftdeps files' paths to the
compiler somehow. We can't figure out how to do this, so this is the point
where our incremental compile fails:
https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348

This is as far as we got. Would super appreciate if anyone on the list
could point us in the right direction from here, especially for the
following questions:

1. This would probably be easier if we could find the right test case. We
poked around in test/Driver/Dependencies which had a bunch of tests around
reading the .swiftdeps files, but we couldn't find tests that demonstrated
how incremental compilation worked at a high level.

2. '-output-file-map': Is this file meant to be written by hand, or is
there a part of the swift compiler that writes this for you?

3. Specifying paths for .swiftdeps: We had assumed this was done based on
the '-output-file-map', but writing the paths we wanted manually did not
seem to work. Any tips?

Thanks so much!
Sam and Brian

On Wed, Apr 13, 2016 at 5:18 PM, Samantha John <sam at gethopscotch.com> wrote:

> Hi Jordan,
>
> The thing that sticks out in the dependency analysis is the treatment of
> external dependencies. The entire module has the same list of external
> dependencies which causes a lot of needless recompiles- especially if you
> start with a large swift project and slowly start to move things into
> modules.
>
> So to me the lowest hanging fruit would be to only mark files for
> recompilation that explicitly import the external dependency. This seems
> pretty safe since you can't compile unless the dependency is explicitly
> imported. Has anyone on the list tried this before?
>
> A second idea would be to consider a file as changed only if its build
> artifact actually changes. Obviously, we'd have to actually build the file
> to figure this out, so we wouldn't have the same level of parallelism
> initially. Perhaps if it was an optional compiler flag this would be more
> palatable? Also wondering if anyone has tried something along these lines.
>
> Thanks!
>
>
> George- The bridge between objective c and swift between is definitely a
> choke point. We've been able to mitigate objective-c recompiles somewhat by
> limiting our imports of swift into objective-c. We've even gone so far as
> to make wrapper classes in objective c around some of our most commonly
> used swift classes so as not to import swift.
>
> It's also very true that changing an objective c .h file that is imported
> into the bridging header will trigger massive recompiles. As more of our
> app has transitioned to swift this has been less of an issue. Most of the
> problems at this point have to do with recompiling a large portion of our
> swift code due to small changes in unrelated parts of our other swift code.
>
>
>
>
>
> Get the latest from Hopscotch!
>
> Sign-up for our newsletter <http://eepurl.com/Ui0eX>
>
> On Fri, Apr 8, 2016 at 5:34 PM, George King <gwk.lists at gmail.com> wrote:
>
>> Hey Sam,
>>
>> One thought: if you have an app with mixed objc and swift code, then the
>> app-bridge.h and app-swift.h files might be creating massive choke points
>> in your dependency graph. I have no idea how optimized the bridging
>> functionality is but it has always seemed like a potentially weak part of
>> the dependency management. I imagine that with an objc half and a swift
>> half of the code base, every time you make a change in a swift file you
>> trigger recompilation of the objc half, and vice versa. I'd love to hear
>> from the core team to what extent this is true!
>>
>> George
>>
>>
>>
>> On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <
>> swift-dev at swift.org> wrote:
>>
>> Thank you Jordan! This is a great starting off point.
>>
>> I'm thinking about proposing a "strict import" mode in swift: A compile
>> flag that when turned on would require you to explicitly import any file
>> that contained a dependency you needed (like in objective-c).
>>
>> I'm going to spend more time looking over the docs and the output logs to
>> see if this would be a feasible. If anyone has opinions or insights into
>> this I would love to hear from you.
>>
>> Sam
>>
>> On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose <jordan_rose at apple.com>
>> wrote:
>>
>>> Hi, Sam. I don't think we currently have a good answer for this built
>>> into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds
>>> would be fast enough that it wouldn't matter! That's obviously not where we
>>> are.)
>>>
>>> Since '-debug-time-function-bodies' is now public knowledge, I'll share
>>> another one of our debugging flags, '-driver-show-incremental'. You can add
>>> this to your "Other Swift Flags". The output isn't very detailed, though:
>>>
>>> Queuing Tree.swift (initial)
>>> Queuing AdventureScene.swift (initial)
>>> Queuing AdventureScene.swift because of dependencies discovered later
>>> Queuing AppDelegate.swift because of dependencies discovered later
>>> Queuing ChaseArtificialIntelligence.swift because of dependencies
>>> discovered later
>>> Queuing Character.swift because of dependencies discovered later
>>> Queuing SpawnArtificialIntelligence.swift because of dependencies
>>> discovered later
>>> Queuing Goblin.swift because of dependencies discovered later
>>> Queuing Cave.swift because of dependencies discovered later
>>> Queuing AdventureSceneOSXEvents.swift because of dependencies discovered
>>> later
>>> Queuing HeroCharacter.swift because of dependencies discovered later
>>> Queuing EnemyCharacter.swift because of dependencies discovered later
>>> Queuing Boss.swift because of dependencies discovered later
>>> Queuing SharedAssetManagement.swift because of dependencies discovered
>>> later
>>> Queuing Warrior.swift because of dependencies discovered later
>>> Queuing Archer.swift because of dependencies discovered later
>>> Queuing Player.swift because of dependencies discovered later
>>> Queuing ArtificialIntelligence.swift because of dependencies discovered
>>> later
>>>
>>> In this case, I took a version of the Adventure sample project and
>>> modified "Tree.swift"; that triggered recompilation of several other files.
>>> Unfortunately this view doesn't tell you how they're related, only which
>>> ones are actually getting rebuilt.
>>>
>>> The next step (and moving into the territory of "working on Swift"
>>> rather than just "trying to figure out why it's repeating work") would be
>>> to look at the "swiftdeps" files stored in your DerivedData folder. These
>>> are currently just YAML files describing what Swift thinks the file depends
>>> on, as well as what will trigger rebuilding of other files. This is
>>> intended to be a conservative estimate, since *not* recompiling
>>> something would result in an invalid binary. (Unfortunately I say
>>> "intended" because there are known bugs; fortunately, archive builds are
>>> always clean builds anyway.)
>>>
>>> There's a document in the Swift repo describing the logic behind Swift's
>>> dependency analysis:
>>> https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst.
>>> The one thing that's *not* in there is the notion of changes that don't
>>> affect other files at all. This is accomplished by computing a hash of all
>>> the tokens that *could* affect other files, and seeing if that hash has
>>> changed.
>>>
>>> We definitely have room for improvement here.
>>>
>>> Jordan
>>>
>>>
>>> On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <
>>> swift-dev at swift.org> wrote:
>>>
>>> I have a large project (308 swift files, 441 objective c, 66k lines of
>>> code) where incremental builds can be extremely slow. I'm trying to do some
>>> profiling to figure out what type of things cause large scale recompiles.
>>> The problem is that I can't find a good way of telling which files get
>>> recompiled on an incremental build and which do not. It seems like files
>>> that are not recompiled still get listed in xcode, but the compiler just
>>> passes over them really fast.
>>>
>>> Does anyone know if xctool or xcodebuild has this type of functionality?
>>> Or is there some other way to get this info?
>>>
>>> Thank you,
>>> Sam
>>> _______________________________________________
>>> swift-dev mailing list
>>> swift-dev at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-dev
>>>
>>>
>>>
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-dev
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160424/ee572ac1/attachment.html>

More information about the swift-dev mailing list