<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Mishal,<div class=""><br class=""></div><div class="">Douglas Gregor suggested in SR-4814 (that was merged as dupe into SR-4669) that I get in contact with you because I’m working on issue <a href="https://bugs.swift.org/browse/SR-4669" class="">SR-4669 Add a Benchmark_Driver --rerun N option</a> and I should make sure that I don’t break the Swift-CI.</div><div class=""><br class=""></div><div class="">The point of that is to deal with false changes (regressions and improvements) that are often seen in benchmark reports from CI.</div><div class=""><br class=""></div><div class="">I have run Benchmark_O with --num-iters=100 on my machine for the the whole performance test suite, to get a feeling for the distribution of benchmark samples, because I also want to move the Benchmark_Driver to use MEAN instead of MIN in the analysis.</div><div class=""><br class=""></div><div class="">There was a tremendous difference on the quality of measured samples depending on how busy my machine was. When I had idle browser windows open in the background, or Xcode open, the results had very high variance. When I purged all I could the results were very stable.</div><div class=""><br class=""></div><div class="">First I want to check with you that the Macs that run the CI and benchmarks do have turned off the Spotlight indexing on the folder that they check out the project into (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just making sure that `mds`, `mds_stores` and `mds_worker` processes don’t unnecessarily impact the benchmark runs. How about Time Machine's `backupd`?</div><div class=""><br class=""></div><div class="">---</div><div class=""><br class=""></div><div class="">Also I have recently filed <a href="https://bugs.swift.org/browse/SR-4659" class="">SR-4659 Benchmark logs should be tied to tested tree version</a> and implemented a fix that got merged in <a href="https://github.com/apple/swift/pull/8978" class="">PR #8978</a>, adding short hash of the commit to the Benchmark log file name. But that works only while you don’t make local changes. In the end I would like to have fully identified versions of benchmarks in the comparison report. In order to make that work, I’ve filed <a href="https://bugs.swift.org/browse/SR-4675" class="">SR-4675 Report detailed build version from Benchmark drivers</a>, that calls for having something akin to `swift —version` be baked into the Benchmark_O{Onone, Ounchecked}. </div><div class=""><br class=""></div><div class="">Given we are on git now, I think the solution is to modify cmake to run `git describe` and generate some kind of version file that will be included during build and reported by the driver when running benchmarks.</div><div class=""><br class=""></div><div class="">I thought it would be nice touch to report also the branch names like one that is cited in the PRs on github. For example:</div><div class=""><br class=""></div><div class="">apple:master vs palimondo:SR-4659</div><div class=""></div><div class=""><br class=""></div><div class="">I found that <a href="https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73" class="">getSVN.cmake</a> from llvm has all the commands to extract the needed information… But, whenI looked at full console logs from swift-ci to see if that would work with the way how the branches are checked out on the build machine, I’m a bit confused:</div><div class=""><br class=""></div><div class=""><blockquote type="cite" class="">Cloning the remote Git repository</blockquote><blockquote type="cite" class="">Cloning repository <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git</blockquote><blockquote type="cite" class=""> > git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # timeout=10</blockquote><blockquote type="cite" class="">Fetching upstream changes from <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git</blockquote><blockquote type="cite" class=""> > git --version # timeout=10</blockquote><blockquote type="cite" class=""> > git fetch --tags --progress <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git +refs/heads/*:refs/remotes/origin/*</blockquote><blockquote type="cite" class=""> > git config remote.origin.url <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git # timeout=10</blockquote><blockquote type="cite" class=""> > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10</blockquote><blockquote type="cite" class=""> > git config remote.origin.url <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git # timeout=10</blockquote><blockquote type="cite" class="">Cleaning workspace</blockquote><blockquote type="cite" class=""> > git rev-parse --verify HEAD # timeout=10</blockquote><blockquote type="cite" class="">No valid HEAD. Skipping the resetting</blockquote><blockquote type="cite" class=""> > git clean -fdx # timeout=10</blockquote><blockquote type="cite" class="">Fetching upstream changes from <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git</blockquote><blockquote type="cite" class=""> > git fetch --tags --progress <a href="mailto:git@github.com" class="">git@github.com</a>:apple/swift.git +refs/pull/*:refs/remotes/origin/pr/*</blockquote><blockquote type="cite" class=""> > git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10</blockquote><blockquote type="cite" class=""> > git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # timeout=10</blockquote><blockquote type="cite" class="">Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 (refs/remotes/origin/pr/9330/merge)</blockquote><blockquote type="cite" class=""> > git config core.sparsecheckout # timeout=10</blockquote><blockquote type="cite" class=""> > git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5</blockquote><blockquote type="cite" class="">Using 'Changelog to branch' strategy.</blockquote><blockquote type="cite" class="">[swift-PR-osx-perf] $ /bin/bash -e /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh</blockquote><blockquote type="cite" class="">Cloning into 'ninja'...</blockquote><blockquote type="cite" class="">Branch: master</blockquote><blockquote type="cite" class="">Found related pull requests: []</blockquote></div><div class=""><br class=""></div><div class="">I don’t quite understand how is the PR branch cloned from github. It definitely looks different from the situation wehn I build on my local machine. </div><div class=""><br class=""></div><div class="">What would be the way to extract remote repository and branch name for the PR being merged?</div><div class=""><br class=""></div><div class="">When I’ll be working on these changes, It would be able to build just the benchmark suite, but <a href="https://bugs.swift.org/browse/SR-4593" class="">SR-4593 Following instructions to independently build benchmarks fails to do so</a>. Who can help me here? (So far I have always been using <span style="color: rgb(101, 123, 131); font-family: 'DejaVu Sans Mono'; background-color: rgba(253, 246, 227, 0.960784);" class="">./swift/utils/build-script -R</span>, but that always takes ages, and often rebuilds whole compiler infrastructure.)</div><div class=""><br class=""></div><div class="">Best regards</div><div class="">Pavol Vaskovic</div><div class=""><br class=""></div><div class=""><br class=""></div></div></body></html>