[swift-dev] incremental compilation

[Erik Eckstein]

I'd like to give some additional information about my recent commit https://github.com/apple/swift/commit/aaaf36e83521f153ba4b0720795efe4980d9b124
The idea is to run the swiftc's llvm-pipeline only if the llvm IR (the output of IRGen) did change. The llvm pipeline takes about 45% to 60% of total compilation time.
It's done by storing a MD5 hash of the IR in the generated object files. Specifically, on macho-o the hash is stored in a special section in the __LLVM segment, which is ignored by the linker.

The intention is to speed-up whole module compilation. We are already running the llvm part multi-threaded, which gives us about 30% speedup with 4 cores. Incremental gives us another 15% to 20%, assuming only a "small" thing changed in one of many files.
Of course the incremental speed-up totally depends on what changes are done on the sources. In worst case a small change might trigger a complete re-compilation e.g. if it's in an often inlined function. 

Here are some numbers for three different modules: a 64 file module which quite large files, a 500 file module with smaller files and the Adventure project. The percentages are similar for all three modules. 



A nice side-effect of incremental compilation is that it works for every type of compilation which produces an object file (not only for multi-threaded wmo). For example if you make changes in the compiler which do not affect code generation, then a library build (ninja swift-stdlib) is faster by 60%.



One additional note about the hash. It also includes the full compiler version, which contains e.g. the git SHAs (in a development build). So every time the branch head changes, everything is recompiled.
The hash also includes some option settings, like -disable-llvm-optzns.
There is only one situation where you should think of incremental compilation: if you make some local changes in swiftc's llvm-pipeline you should explicitly clean all compiled libraries, etc.

Erik

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160129/57f82f6b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 27184 bytes
Desc: not available
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160129/57f82f6b/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-2.png
Type: image/png
Size: 25402 bytes
Desc: not available
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160129/57f82f6b/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-3.png
Type: image/png
Size: 26582 bytes
Desc: not available
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160129/57f82f6b/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-5.png
Type: image/png
Size: 21264 bytes
Desc: not available
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20160129/57f82f6b/attachment-0003.png>