<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Jul 14, 2017, at 4:15 PM, Charles Srstka <<a href="mailto:cocoadev@charlessoft.com" class="">cocoadev@charlessoft.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><blockquote type="cite" class="">On Jul 14, 2017, at 2:35 PM, John McCall <<a href="mailto:rjmccall@apple.com" class="">rjmccall@apple.com</a>> wrote:<br class=""></blockquote><div class=""><blockquote type="cite" class=""><br class="Apple-interchange-newline"><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div class=""><blockquote type="cite" class="" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><div class="">On Jul 14, 2017, at 1:12 PM, Charles Srstka via swift-evolution <<a href="mailto:swift-evolution@swift.org" class="">swift-evolution@swift.org</a>> wrote:</div><div class=""><div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">MOTIVATION:<div class=""><br class=""></div><div class="">Meet Bob. Bob is a developer with mostly C++ and Java experience, but who has been learning Swift. Bob needs to write an app to parse some proprietary binary data format that his company requires. Bob’s written this app, and it’s worked pretty well on Linux:</div><div class=""><br class=""></div><div class=""><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"><span class="" style="color: rgb(186, 45, 162);">import</span><span class="Apple-converted-space"> </span>Foundation</div><div class="" style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255); min-height: 14px;"><br class=""></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(186, 45, 162); background-color: rgb(255, 255, 255);">do<span class=""><span class="Apple-converted-space"> </span>{</span></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"><span class=""> <span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(186, 45, 162);">let</span><span class=""><span class="Apple-converted-space"> </span>url =<span class="Apple-converted-space"> </span></span><font color="#703daa" class="">...</font></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">let</span><span class="Apple-converted-space"> </span>handle =<span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">try</span><span class="Apple-converted-space"> </span><span class="" style="color: rgb(112, 61, 170);">FileHandle</span>(forReadingFrom: url)</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); background-color: rgb(255, 255, 255);"><span class=""> <span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(186, 45, 162);">let</span><span class=""><span class="Apple-converted-space"> </span>bufsize =<span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(39, 42, 216);">1024</span><span class=""><span class="Apple-converted-space"> </span>*<span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(39, 42, 216);">1024</span><span class=""><span class="Apple-converted-space"> </span></span>// read 1 MiB at a time</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(186, 45, 162); background-color: rgb(255, 255, 255);"><span class=""> <span class="Apple-converted-space"> </span></span>while<span class=""><span class="Apple-converted-space"> </span></span>true<span class=""><span class="Apple-converted-space"> </span>{</span></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">let</span><span class="Apple-converted-space"> </span>data = handle.<span class="" style="color: rgb(62, 30, 129);">readData</span>(ofLength: bufsize)</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">if</span><span class="Apple-converted-space"> </span>data.<span class="" style="color: rgb(112, 61, 170);">isEmpty</span><span class="Apple-converted-space"> </span>{</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">break</span></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> }</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255); min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> data.<span class="" style="color: rgb(62, 30, 129);">withUnsafeBytes</span><span class="Apple-converted-space"> </span>{ (bytes:<span class="Apple-converted-space"> </span><span class="" style="color: rgb(112, 61, 170);">UnsafePointer</span><<span class="" style="color: rgb(112, 61, 170);">UInt8</span>>)<span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">in</span></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(0, 132, 0); background-color: rgb(255, 255, 255);"><span class=""> <span class="Apple-converted-space"> </span></span>// do something with bytes</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> }</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"> }</div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(186, 45, 162); background-color: rgb(255, 255, 255);"><span class="">}<span class="Apple-converted-space"> </span></span>catch<span class=""><span class="Apple-converted-space"> </span>{</span></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; color: rgb(112, 61, 170); background-color: rgb(255, 255, 255);"><span class=""> <span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(62, 30, 129);">print</span><span class="">(</span><span class="" style="color: rgb(209, 47, 27);">"Error occurred:<span class="Apple-converted-space"> </span></span><span class="">\</span><span class="" style="color: rgb(209, 47, 27);">(</span><span class="">error.</span>localizedDescription<span class="" style="color: rgb(209, 47, 27);">)"</span><span class="">)</span></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);">}</div></div><div class=""><br class=""></div><div class="">Later, Bob needs to port this same app to macOS. All seems to work well, until Bob tries opening a large file of many gigabytes in size. Suddenly, the simple act of running the app causes Bob’s Mac to completely lock up, beachball, and finally pop up with the dreaded “This computer is out of system memory” message. If Bob’s particularly unlucky, things will locked up tight enough that he can’t even recover from there, and may have to hard-reboot the machine.</div><div class=""><br class=""></div><div class="">What happened?</div><div class=""><br class=""></div><div class="">Experienced Objective-C developers will spot the problem right away; the Foundation APIs that Bob used generated autoreleased objects, which would never be released until Bob’s loop finished. However, Bob’s never programmed in Objective-C, and to him, this behavior is completely undecipherable.</div><div class=""><br class=""></div><div class="">After a copious amount of time spent Googling for answers and asking for help on various mailing lists and message boards, Bob finally gets the recommendation from someone to try wrapping the file handle read in an autorelease pool. So he does:</div><div class=""><br class=""></div><div class=""><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"><div class="" style="margin: 0px; line-height: normal;"><span class="" style="color: rgb(186, 45, 162);">import</span><span class="Apple-converted-space"> </span>Foundation</div><div class="" style="margin: 0px; font-size: 12px; line-height: normal; font-family: Helvetica; min-height: 14px;"><br class=""></div><div class="" style="margin: 0px; line-height: normal; color: rgb(186, 45, 162);">do<span class=""><span class="Apple-converted-space"> </span>{</span></div><div class="" style="margin: 0px; line-height: normal;"><span class=""> <span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(186, 45, 162);">let</span><span class=""><span class="Apple-converted-space"> </span>url =<span class="Apple-converted-space"> </span></span><font color="#703daa" class="">...</font></div><div class="" style="margin: 0px; line-height: normal; min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; line-height: normal;"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">let</span><span class="Apple-converted-space"> </span>handle =<span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">try</span><span class="Apple-converted-space"> </span><span class="" style="color: rgb(112, 61, 170);">FileHandle</span>(forReadingFrom: url)</div><div class="" style="margin: 0px; line-height: normal; color: rgb(0, 132, 0);"><span class=""> <span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(186, 45, 162);">let</span><span class=""><span class="Apple-converted-space"> </span>bufsize =<span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(39, 42, 216);">1024</span><span class=""><span class="Apple-converted-space"> </span>*<span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(39, 42, 216);">1024</span><span class=""><span class="Apple-converted-space"> </span></span>// read 1 MiB at a time</div><div class="" style="margin: 0px; line-height: normal; min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; line-height: normal; color: rgb(186, 45, 162);"><span class=""> <span class="Apple-converted-space"> </span></span>while<span class=""><span class="Apple-converted-space"> </span></span>true<span class=""><span class="Apple-converted-space"> </span>{</span></div><div class="" style="margin: 0px; line-height: normal;"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">let</span><span class="Apple-converted-space"> </span>data =<span class="Apple-converted-space"> </span><span class="" style="color: rgb(62, 30, 129);">autoreleasepool</span><span class="Apple-converted-space"> </span>{ handle.<span class="" style="color: rgb(62, 30, 129);">readData</span>(ofLength: bufsize) }</div><div class="" style="margin: 0px; line-height: normal; min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; line-height: normal;"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">if</span><span class="Apple-converted-space"> </span>data.<span class="" style="color: rgb(112, 61, 170);">isEmpty</span><span class="Apple-converted-space"> </span>{</div><div class="" style="margin: 0px; line-height: normal;"> <span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">break</span></div><div class="" style="margin: 0px; line-height: normal;"> }</div><div class="" style="margin: 0px; line-height: normal; min-height: 16px;"> <br class="webkit-block-placeholder"></div><div class="" style="margin: 0px; line-height: normal;"> data.<span class="" style="color: rgb(62, 30, 129);">withUnsafeBytes</span><span class="Apple-converted-space"> </span>{ (bytes:<span class="Apple-converted-space"> </span><span class="" style="color: rgb(112, 61, 170);">UnsafePointer</span><<span class="" style="color: rgb(112, 61, 170);">UInt8</span>>)<span class="Apple-converted-space"> </span><span class="" style="color: rgb(186, 45, 162);">in</span></div><div class="" style="margin: 0px; line-height: normal; color: rgb(0, 132, 0);"><span class=""> <span class="Apple-converted-space"> </span></span>// do something with bytes</div><div class="" style="margin: 0px; line-height: normal;"> }</div><div class="" style="margin: 0px; line-height: normal;"> }</div><div class="" style="margin: 0px; line-height: normal; color: rgb(186, 45, 162);"><span class="">}<span class="Apple-converted-space"> </span></span>catch<span class=""><span class="Apple-converted-space"> </span>{</span></div><div class="" style="margin: 0px; line-height: normal; color: rgb(112, 61, 170);"><span class=""> <span class="Apple-converted-space"> </span></span><span class="" style="color: rgb(62, 30, 129);">print</span><span class="">(</span><span class="" style="color: rgb(209, 47, 27);">"Error occurred:<span class="Apple-converted-space"> </span></span><span class="">\</span><span class="" style="color: rgb(209, 47, 27);">(</span><span class="">error.</span>localizedDescription<span class="" style="color: rgb(209, 47, 27);">)"</span><span class="">)</span></div><div class="" style="margin: 0px; line-height: normal;">}</div></div></div><div class="" style="margin: 0px; font-size: 14px; line-height: normal; font-family: Menlo; background-color: rgb(255, 255, 255);"><br class=""></div><div class="" style="margin: 0px; line-height: normal; background-color: rgb(255, 255, 255);">Unfortunately, Bob’s program still eats RAM like Homer Simpson in an all-you-can-eat buffet. Turns out the data.withUnsafeBytes call *also* causes the data to be autoreleased.</div></div></div></blockquote><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""></div><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">This seems like a bug that should be fixed. I don't know why the other one would cause an unreclaimable autorelease.</div></div></div></div></blockquote><br class=""></div>Sticking a break at the end of my original code snippet so the loop runs only once and then running it through Instruments, it seems the NSConcreteData instance gets autoreleased… 32,769 times. o_O<div class=""><br class=""></div><div class="">First autorelease occurs inside -[NSConcreteFileHandle readDataOfLength:], the next 32,768 occur in Data.Iterator.next(), which is called by specialized RangeReplaceableCollection.init<A>.</div><div class=""><br class=""></div><div class="">I can send you the trace off-list if you’d like.</div></div></div></blockquote><div><br class=""></div></div>We should absolutely not need to do the later autoreleases. We have logic to autorelease objects when calling returns-inner-pointer objects on them, but we shouldn't need to do that in safe patterns like what Data does here by scoping the pointer to the closure. We probably just don't actually have a way to turn that logic off, i.e. an equivalent of objc_precise_lifetime in ObjC ARC.<div class=""><br class=""></div><div class="">I have no idea why the first autorelease wouldn't be reclaimed. There's a well-known issue with micro-reductions involving autoreleases on x86, where the first autorelease from the executable doesn't get reclaimed because the dynamic linker's lazy-binding stub interferes somehow. Can you verify that you still see that initial autorelease on subsequent Data creations?</div><div class=""><br class=""></div><div class="">John.</div></body></html>