[swift-dev] Mysterious memory improvement and regression

[Pavol Vaskovic]

Greg!  

Thanks for the explanation and sorry for the report. I’m putting the attachment to dropbox, to work around the mailing list size cap.  
> On 25 Apr 2017, at 08:36, Greg Parker <gparker at apple.com (mailto:gparker at apple.com)> wrote:
>  
> > On Apr 24, 2017, at 8:52 PM, Pavol Vaskovic <pali at pali.sk (mailto:pali at pali.sk)> wrote:
> >  
> > > On 25 Apr 2017, at 01:28, Greg Parker <gparker at apple.com (mailto:gparker at apple.com)> wrote:
> > >  
> > > The value of MAX_RSS depends on OS behavior. Other activity on the same machine may change MAX_RSS of the benchmark.
> > Can you please describe the mechanism of how “other activity on the same machine may change MAX_RSS of the benchmark”.?
> Not all memory used by the benchmark counts against its RSS. For example, paged-out and VM-compressed memory are both excluded. If there are other processes contending for memory while the benchmark runs then the benchmark's RSS will be artificially reduced.
> Did you see any time difference between the 3MB and the 10MB runs?  

Yes. A negative inverse. The runs with 10 MB were mostly faster. 3MB were mostly slower, but it differed between various benchmark families - some were comparable speed.
>  
> > > The changes you saw might not be "real”.
> > How was that not real? I have logs that prove that. See the attachment in original post.
> The possibility is that the benchmark's RSS did in fact change, but the benchmark's "real" memory usage did not.
OK, I understand. Would you say its possible I did one of those measurements under extreme memory pressure on my machine and that’s what caused this? The so the Swift runtime gets compacted under low memory conditions with memory compression from 10 to 3 MB? And correspondingly tests that need to access compressed parts are slowed down by decompression and some lucky tests evade this penalty.

>  
> > > If you re-run the 2017-04-15 builds today, do you still see 3 MB instead of 10 MB?
> > Nope. If I saw it now, I wouldn’t be searching for that, but congratulating the brave committer that gave us this improvement.  
>  
> That tends to suggest a difference in the test environment rather than a change in Swift. If some Swift changes were responsible for the RSS decrease and subsequent increase then re-running the benchmark with that version of Swift ought to exhibit the same RSS behavior.


The problem is that I have notices this after the fact, few days passed and I don’t know which revision was tested - benchmarks didn’t log it then. I’ve filed these issue because of it:
* https://bugs.swift.org/browse/SR-4659 Benchmark logs should be tied to tested tree version
* https://bugs.swift.org/browse/SR-4675 Report detailed build version from Benchmark drivers​

> Measuring memory usage is good. RSS is a difficult value to use for such measurements.Please have a look at the Numbers file at https://db.tt/CnhRTCrqED. The 2 tabs of interest are Release-3s-1 from 20170414 (ie. before) and Release-3s from 20170415 (the change). Later measurements returned to the before value. Last 4 columns are the differences between the max RSS and median time, with percentages (color coded).

Best regards
Pavol Vaskovic