<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 25, 2016, at 8:15 PM, Samantha John <<a href="mailto:sam@gethopscotch.com" class="">sam@gethopscotch.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote">On Mon, Apr 25, 2016 at 12:11 AM, Daniel Dunbar<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:daniel_dunbar@apple.com" target="_blank" class="">daniel_dunbar@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><br class=""><div class=""><span class=""><blockquote type="cite" class=""><div class="">On Apr 24, 2016, at 3:19 PM, Samantha John via swift-dev <<a href="mailto:swift-dev@swift.org" target="_blank" class="">swift-dev@swift.org</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class="">Hello List (cc/Jordan), </div><div class=""><br class=""></div><div class="">At a high level: Brian and I are looking into contributing to incremental compilation in Swift. Right now we're trying to do an incremental compile by invoking swiftc. </div><div class=""><br class=""></div><div class="">Our understanding so far:</div><div class="">- swiftc takes a list of input files. </div><div class="">- If swiftc is invoked with '-incremental', it requires an '-output-file-map' option to also be passed. We assume this is used to determine where intermediate dependency files go.</div><div class="">- Looking at the tests in test/Driver/Dependencies/Inputs and the sorts of files Xcode generates when building Swift projects, we determined '-output-file-map' to be a JSON file (which is parsed by the llvm yaml parser?<span class="Apple-converted-space"> </span><a href="https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100" target="_blank" class="">https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/OutputFileMap.cpp#L100</a>). We tried writing our own, this is the shortest version that the compiler didn't error on:</div><div class=""><br class=""></div><div class="">```</div><div class="">{</div><div class=""> <span class="Apple-converted-space"> </span>"": {</div><div class=""> <span class="Apple-converted-space"> </span>"swift-dependencies": "MyModule-main.swiftdeps"</div><div class=""> <span class="Apple-converted-space"> </span>}</div><div class="">}</div><div class="">```</div><div class=""><br class=""></div><div class="">- When compiling with the above file ('swiftc -incremental -output-file-map OurFileMap.json *.swift'), swiftc writes to MyModule-main.swiftdeps. The end result looks like this:</div><div class=""><br class=""></div><div class="">```</div><div class="">version: "Swift version 3.0-dev (LLVM 752e1430fc, Clang 3987718dae, Swift a2cf18ba2f)"</div><div class="">options: "9277a78155e85019ce36a3c52e9f3f02"</div><div class="">build_time: [514847765, 412105000]</div><div class="">inputs:</div><div class=""> <span class="Apple-converted-space"> </span>"Class1.swift": [514841531, 0]</div><div class=""> <span class="Apple-converted-space"> </span>"Class2.swift": [514844635, 0]</div><div class=""> <span class="Apple-converted-space"> </span>"main.swift": [514841821, 0]</div><div class="">```</div><div class=""><br class=""></div><div class="">- Where the [xxx, yyy] are timestamps with the first number representing seconds, the second nanoseconds. (<a href="https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221" target="_blank" class="">https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Driver.cpp#L221</a>)</div><div class=""><br class=""></div><div class="">- We invoked 'swiftc -incremental -output-file-map OurFileMap.json *.swift -parseable-output -save-temps' to show us the paths to the generated .swiftdeps files. We assume that to get incremental compiles to work for us, we'd need to pass these generated .swiftdeps files' paths to the compiler somehow. We can't figure out how to do this, so this is the point where our incremental compile fails:<span class="Apple-converted-space"> </span><a href="https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348" target="_blank" class="">https://github.com/apple/swift/blob/95e3be665d9387eb0d354f3d0f313e44c7b4245d/lib/Driver/Compilation.cpp#L348</a></div></div></div></blockquote><div class=""><br class=""></div></span>You don't need to point the compiler at the swiftdeps files, it reuses the ones passed in the output file map if -incremental is passed.</div><div class=""><br class=""></div><div class="">You didn't mention it (I don't think) but you also need to pass -emit-dependencies, however it sounds like you are doing this.</div><div class=""><span class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="">This is as far as we got. Would super appreciate if anyone on the list could point us in the right direction from here, especially for the following questions:</div><div class=""><br class=""></div><div class="">1. This would probably be easier if we could find the right test case. We poked around in test/Driver/Dependencies which had a bunch of tests around reading the .swiftdeps files, but we couldn't find tests that demonstrated how incremental compilation worked at a high level.</div><div class=""><br class=""></div><div class="">2. '-output-file-map': Is this file meant to be written by hand, or is there a part of the swift compiler that writes this for you?</div></div></div></blockquote><div class=""><br class=""></div></span>It is meant to be written by the build system which invokes Swift. See also:</div><div class=""> <a href="https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206" target="_blank" class="">https://github.com/apple/swift-llbuild/blob/master/lib/BuildSystem/SwiftTools.cpp#L206</a></div><div class="">which is what the Swift package manager uses (which supports incremental compiles).</div></div></blockquote><div class=""><br class=""></div><div class="">This sounds perfect. I tried to install the llbuild but got stuck on filecheck (and saw that you are trying to add it to the toolchain:<span class="Apple-converted-space"> </span><a href="https://github.com/apple/swift-llbuild/pull/22/files#r61007286" class="">https://github.com/apple/swift-llbuild/pull/22/files#r61007286</a>) Is there an executable of llbuild I could download, or do you have any basic instructions for what to do about file check? </div></div></div></div></div></blockquote><div><br class=""></div>You shouldn't need to download llbuild, you probably want to go the xcodebuild route...</div><div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""></div><div class="">Your best bet is to take the exact command line used by xcodebuild, and then invoke swiftc with that to replicate the incremental build. If you run with -v you should be able to see exactly what files get built </div></div></blockquote><div class=""><br class=""></div><div class=""> I tried running xcodebuild with the verbose flag but wasn't getting much useful. Is that what you meant here? -v didn't seem to be an option. </div></div></div></div></div></blockquote><div><br class=""></div>I wasn't very clear here. What you need to do is run xcodebuild from the command line, and then find the top-level line which is compiling the module you are about (this will be the line with `swiftc ... -incremental ...` in it but not including `-frontend). That is the command you can run with `-v`.</div><div><br class=""></div><div> - Daniel</div><div><br class=""></div><div><blockquote type="cite" class=""><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="gmail_extra"><div class="gmail_quote"><div class=""> </div><div class="">Thanks so much, <br class="">Sam<br class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""></div><div class="">HTH!</div><span class=""><font color="#888888" class=""><div class=""> - Daniel</div></font></span><div class=""><div class=""><div class=""><br class=""></div><div class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class=""></div><div class="">3. Specifying paths for .swiftdeps: We had assumed this was done based on the '-output-file-map', but writing the paths we wanted manually did not seem to work. Any tips?</div><div class=""><br class=""></div><div class="">Thanks so much!</div><div class="">Sam and Brian</div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 13, 2016 at 5:18 PM, Samantha John<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:sam@gethopscotch.com" target="_blank" class="">sam@gethopscotch.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div dir="ltr" class=""><div class="">Hi Jordan, </div><div class=""><br class=""></div><div class="">The thing that sticks out in the dependency analysis is the treatment of external dependencies. The entire module has the same list of external dependencies which causes a lot of needless recompiles- especially if you start with a large swift project and slowly start to move things into modules. </div><div class=""><br class=""></div><div class="">So to me the lowest hanging fruit would be to only mark files for recompilation that explicitly import the external dependency. This seems pretty safe since you can't compile unless the dependency is explicitly imported. Has anyone on the list tried this before? </div><div class=""><br class=""></div><div class="">A second idea would be to consider a file as changed only if its build artifact actually changes. Obviously, we'd have to actually build the file to figure this out, so we wouldn't have the same level of parallelism initially. Perhaps if it was an optional compiler flag this would be more palatable? Also wondering if anyone has tried something along these lines.</div><div class=""><br class=""></div><div class="">Thanks! </div><div class=""><br class=""></div><div class=""><br class=""></div>George- The bridge between objective c and swift between is definitely a choke point. We've been able to mitigate objective-c recompiles somewhat by limiting our imports of swift into objective-c. We've even gone so far as to make wrapper classes in objective c around some of our most commonly used swift classes so as not to import swift. <div class=""><br class=""></div><div class="">It's also very true that changing an objective c .h file that is imported into the bridging header will trigger massive recompiles. As more of our app has transitioned to swift this has been less of an issue. Most of the problems at this point have to do with recompiling a large portion of our swift code due to small changes in unrelated parts of our other swift code. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class=""><div dir="ltr" class=""><p class="">Get the latest from Hopscotch!</p><p class=""><span class=""><a href="http://eepurl.com/Ui0eX" target="_blank" class="">Sign-up for our newsletter</a></span></p></div></div></div><div class=""><div class=""><br class=""><div class="gmail_quote">On Fri, Apr 8, 2016 at 5:34 PM, George King<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:gwk.lists@gmail.com" target="_blank" class="">gwk.lists@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class="">Hey Sam,<div class=""><br class=""></div><div class="">One thought: if you have an app with mixed objc and swift code, then the app-bridge.h and app-swift.h files might be creating massive choke points in your dependency graph. I have no idea how optimized the bridging functionality is but it has always seemed like a potentially weak part of the dependency management. I imagine that with an objc half and a swift half of the code base, every time you make a change in a swift file you trigger recompilation of the objc half, and vice versa. I'd love to hear from the core team to what extent this is true!</div><span class=""><font color="#888888" class=""><div class=""><br class=""></div><div class="">George</div></font></span><div class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 7, 2016, at 5:35 PM, Samantha John via swift-dev <<a href="mailto:swift-dev@swift.org" target="_blank" class="">swift-dev@swift.org</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Thank you Jordan! This is a great starting off point.<div class=""><br class=""></div><div class="">I'm thinking about proposing a "strict import" mode in swift: A compile flag that when turned on would require you to explicitly import any file that contained a dependency you needed (like in objective-c).</div><div class=""><br class=""></div><div class="">I'm going to spend more time looking over the docs and the output logs to see if this would be a feasible. If anyone has opinions or insights into this I would love to hear from you. </div><div class=""><br class=""></div><div class="">Sam</div><div class=""><br class=""></div><div class="gmail_extra"><div class="gmail_quote">On Tue, Apr 5, 2016 at 9:08 PM, Jordan Rose<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:jordan_rose@apple.com" target="_blank" class="">jordan_rose@apple.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class="">Hi, Sam. I don't think we currently have a good answer for this built into xcodebuild or xctool, and it's a reasonable idea. (Ideally all builds would be fast enough that it wouldn't matter! That's obviously not where we are.)</div><div class=""><br class=""></div><div class="">Since '-debug-time-function-bodies' is now public knowledge, I'll share another one of our debugging flags, '-driver-show-incremental'. You can add this to your "Other Swift Flags". The output isn't very detailed, though:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Tree.swift (initial)</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing AdventureScene.swift (initial)</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing AdventureScene.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing AppDelegate.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing ChaseArtificialIntelligence.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Character.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing SpawnArtificialIntelligence.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Goblin.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Cave.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing AdventureSceneOSXEvents.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing HeroCharacter.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing EnemyCharacter.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Boss.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing SharedAssetManagement.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Warrior.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Archer.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing Player.swift because of dependencies discovered later</span></div><div style="margin: 0px; font-size: 11px; line-height: normal; font-family: Menlo; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);" class=""><span class="">Queuing ArtificialIntelligence.swift because of dependencies discovered later</span></div></div><div class=""><span class=""><br class=""></span></div><div class=""><span class="">In this case, I took a version of the Adventure sample project and modified "Tree.swift"; that triggered recompilation of several other files. Unfortunately this view doesn't tell you how they're related, only which ones are actually getting rebuilt.</span></div><div class=""><span class=""><br class=""></span></div><div class="">The next step (and moving into the territory of "working on Swift" rather than just "trying to figure out why it's repeating work") would be to look at the "swiftdeps" files stored in your DerivedData folder. These are currently just YAML files describing what Swift thinks the file depends on, as well as what will trigger rebuilding of other files. This is intended to be a conservative estimate, since<span class="Apple-converted-space"> </span><i class="">not</i> recompiling something would result in an invalid binary. (Unfortunately I say "intended" because there are known bugs; fortunately, archive builds are always clean builds anyway.)</div><div class=""><br class=""></div><div class="">There's a document in the Swift repo describing the logic behind Swift's dependency analysis: <a href="https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst" target="_blank" class="">https://github.com/apple/swift/blob/master/docs/DependencyAnalysis.rst</a>. The one thing that's<span class="Apple-converted-space"> </span><i class="">not</i> in there is the notion of changes that don't affect other files at all. This is accomplished by computing a hash of all the tokens that<span class="Apple-converted-space"> </span><i class="">could</i> affect other files, and seeing if that hash has changed.</div><div class=""><br class=""></div><div class="">We definitely have room for improvement here.</div><div class=""><br class=""></div><div class="">Jordan</div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class=""><div class=""><div class="">On Mar 31, 2016, at 11:24 , Samantha John via swift-dev <<a href="mailto:swift-dev@swift.org" target="_blank" class="">swift-dev@swift.org</a>> wrote:</div><br class=""></div></div><div class=""><div class=""><div class=""><div dir="ltr" class=""><div style="font-size: 12.8px;" class="">I have a large project (308 swift files, 441 objective c, 66k lines of code) where incremental builds can be extremely slow. I'm trying to do some profiling to figure out what type of things cause large scale recompiles. The problem is that I can't find a good way of telling which files get recompiled on an incremental build and which do not. It seems like files that are not recompiled still get listed in xcode, but the compiler just passes over them really fast. </div><div style="font-size: 12.8px;" class=""><span style="font-size: 12.8px;" class=""><br class=""></span></div><div style="font-size: 12.8px;" class=""><span style="font-size: 12.8px;" class="">Does anyone know if xctool or xcodebuild has this type of functionality? Or is there some other way to get this info?</span><br class=""></div><div style="font-size: 12.8px;" class=""><br class=""></div><div style="font-size: 12.8px;" class="">Thank you,<br class="">Sam</div></div></div></div>_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" target="_blank" class="">swift-dev@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-dev" target="_blank" class="">https://lists.swift.org/mailman/listinfo/swift-dev</a><br class=""></div></blockquote></div><br class=""></div></blockquote></div><br class=""></div></div>_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" target="_blank" class="">swift-dev@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-dev" target="_blank" class="">https://lists.swift.org/mailman/listinfo/swift-dev</a><br class=""></div></blockquote></div><br class=""></div></div></div></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div>_______________________________________________<br class="">swift-dev mailing list<br class=""><a href="mailto:swift-dev@swift.org" target="_blank" class="">swift-dev@swift.org</a><br class=""><a href="https://lists.swift.org/mailman/listinfo/swift-dev" target="_blank" class="">https://lists.swift.org/mailman/listinfo/swift-dev</a></div></blockquote></div></div></div></div></blockquote></div></div></div></div></blockquote></div><br class=""></body></html>