[swift-dev] Questions about Swift-CI

[Dave Abrahams]

on Thu May 11 2017, Pavol Vaskovic <swift-dev-AT-swift.org> wrote:

> Hi Mishal,
>
> Douglas Gregor suggested in SR-4814 (that was merged as dupe into
> SR-4669) that I get in contact with you because I’m working on issue
> SR-4669 Add a Benchmark_Driver --rerun N option
> <https://bugs.swift.org/browse/SR-4669> and I should make sure that I
> don’t break the Swift-CI.
>
> The point of that is to deal with false changes (regressions and improvements) that are often seen
> in benchmark reports from CI.
>
> I have run Benchmark_O with --num-iters=100 on my machine for the the
> whole performance test suite, to get a feeling for the distribution of
> benchmark samples, because I also want to move the Benchmark_Driver to
> use MEAN instead of MIN in the analysis.

I'm concerned about that, especially for microbenchmarks; it seems to me
as though MIN is the right measurement.  Can you explain why MEAN is
better?

> There was a tremendous difference on the quality of measured samples
> depending on how busy my machine was. When I had idle browser windows
> open in the background, or Xcode open, the results had very high
> variance. When I purged all I could the results were very stable.
>
> First I want to check with you that the Macs that run the CI and
> benchmarks do have turned off the Spotlight indexing on the folder
> that they check out the project into
> (/Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift). Just
> making sure that `mds`, `mds_stores` and `mds_worker` processes don’t
> unnecessarily impact the benchmark runs. How about Time Machine's
> `backupd`?
>
> ---
>
> Also I have recently filed SR-4659 Benchmark logs should be tied to
> tested tree version <https://bugs.swift.org/browse/SR-4659> and
> implemented a fix that got merged in PR #8978
> <https://github.com/apple/swift/pull/8978>, adding short hash of the
> commit to the Benchmark log file name. But that works only while you
> don’t make local changes. In the end I would like to have fully
> identified versions of benchmarks in the comparison report. 

Have you thought about using git notes to attach results to commits?
That has always seemed to me like a good way to manage test results.

> In order to make that work, I’ve filed SR-4675 Report detailed build
> version from Benchmark drivers
> <https://bugs.swift.org/browse/SR-4675>, that calls for having
> something akin to `swift —version` be baked into the
> Benchmark_O{Onone, Ounchecked}.
>
> Given we are on git now, I think the solution is to modify cmake to
> run `git describe` and generate some kind of version file that will be
> included during build and reported by the driver when running
> benchmarks.
>
> I thought it would be nice touch to report also the branch names like one that is cited in the PRs
> on github. For example:
>
> apple:master vs palimondo:SR-4659
>
> I found that getSVN.cmake
> <https://github.com/apple/swift-llvm/blob/stable/cmake/modules/GetSVN.cmake#L73>
> from llvm has all the commands to extract the needed information… But,
> whenI looked at full console logs from swift-ci to see if that would
> work with the way how the branches are checked out on the build
> machine, I’m a bit confused:
>
>> Cloning the remote Git repository
>> Cloning repository git at github.com:apple/swift.git
>>  > git init /Users/buildnode/jenkins/workspace/swift-PR-osx-perf/swift # timeout=10
>> Fetching upstream changes from git at github.com:apple/swift.git
>>  > git --version # timeout=10
>>  > git fetch --tags --progress git at github.com:apple/swift.git +refs/heads/*:refs/remotes/origin/*
>>  > git config remote.origin.url git at github.com:apple/swift.git # timeout=10
>>  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
>>  > git config remote.origin.url git at github.com:apple/swift.git # timeout=10
>> Cleaning workspace
>>  > git rev-parse --verify HEAD # timeout=10
>> No valid HEAD. Skipping the resetting
>>  > git clean -fdx # timeout=10
>> Fetching upstream changes from git at github.com:apple/swift.git
>>  > git fetch --tags --progress git at github.com:apple/swift.git +refs/pull/*:refs/remotes/origin/pr/*
>>  > git rev-parse refs/remotes/origin/pr/9330/merge^{commit} # timeout=10
>>  > git rev-parse refs/remotes/origin/origin/pr/9330/merge^{commit} # timeout=10
>> Checking out Revision 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5 (refs/remotes/origin/pr/9330/merge)
>>  > git config core.sparsecheckout # timeout=10
>>  > git checkout -f 55f2b7e24356b31fd29c690d779f9aaf4a3d59d5
>> Using 'Changelog to branch' strategy.
>> [swift-PR-osx-perf] $ /bin/bash -e
> /var/folders/_8/79jmzf2142z2xydc_01btlx00000gn/T/hudson7057602820198606732.sh
>> Cloning into 'ninja'...
>> Branch:   master
>> Found related pull requests: []
>
> I don’t quite understand how is the PR branch cloned from github. It definitely looks different from
> the situation wehn I build on my local machine.
>
> What would be the way to extract remote repository and branch name for the PR being merged?
>
> When I’ll be working on these changes, It would be able to build just
> the benchmark suite, but SR-4593 Following instructions to
> independently build benchmarks fails to do so
> <https://bugs.swift.org/browse/SR-4593>. Who can help me here? (So far
> I have always been using ./swift/utils/build-script -R, but that
> always takes ages, and often rebuilds whole compiler infrastructure.)
>
> Best regards
> Pavol Vaskovic
>
> _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org
> https://lists.swift.org/mailman/listinfo/swift-dev
>

-- 
-Dave