[swift-dev] Inability to leverage privacy in the stdlib

Slava Pestov spestov at apple.com
Tue Nov 1 14:34:42 CDT 2016 

The problem is if a private type or function X is referenced from a function Y with a serialized body, then if Y is inlined inside of another module, it cannot see the symbol for X. The solution is to disable -sil-seiralize-all and replace it with an explicit ‘fragile’ attribute that is put on certain functions only. I’m still working on this but it is almost ready to land.

Slava

> On Nov 1, 2016, at 11:19 AM, Dave Abrahams via swift-dev <swift-dev at swift.org> wrote:
> 
> 
> on Tue Nov 01 2016, Alexis <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/>> wrote:
> 
>> Hmm, I excitedly prototyped out refactoring HashedCollections to
>> properly use privacy and decent naming since I’m clobbering the whole
>> file with the new indexing stuff anyway. Unfortunately I hit a linker
>> error. I can’t seem to produce a reduced test case.
> 
> If you are doing privacy inside the stdlib, then this is similar to the
> experience I recall having the last time I tried to do it.  I'm pretty
> sure that's what the frontend team means when they say that it's
> unsupported.
> 
>> Making similar but smaller changes to other files seems to always
>> work.
>> 
>> The commit of interest is this:
>> 
>> https://github.com/apple/swift/commit/851228a1d59e292b020c16a0fba4561a76f8fec5 <https://github.com/apple/swift/commit/851228a1d59e292b020c16a0fba4561a76f8fec5>
>> <https://github.com/apple/swift/commit/851228a1d59e292b020c16a0fba4561a76f8fec5 <https://github.com/apple/swift/commit/851228a1d59e292b020c16a0fba4561a76f8fec5>>
>> 
>> But it requires all the native indexing stuff I’ve done: 
>> 
>> https://github.com/apple/swift/compare/master...Gankro:dsindex2?expand=1 <https://github.com/apple/swift/compare/master...Gankro:dsindex2?expand=1>
>> <https://github.com/apple/swift/compare/master...Gankro:dsindex2?expand=1 <https://github.com/apple/swift/compare/master...Gankro:dsindex2?expand=1>>
>> 
>> The error I get:
>> 
>> FAILED: : &&
>> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++
>> -fno-stack-protector -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter
>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wcovered-switch-default -Wnon-virtual-dtor
>> -Wdelete-non-virtual-dtor -Werror=date-time -std=c++11 -fcolor-diagnostics -Wdocumentation
>> -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -O2 -isysroot
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk
>> -dynamiclib -Wl,-headerpad_max_install_names -stdlib=libc++ -target x86_64-apple-macosx10.9
>> -isysroot
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk
>> -arch x86_64 -F
>> /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/../../../Developer/Library/Frameworks
>> -mmacosx-version-min=10.9 -lobjc
>> -Wl,-sectcreate,__TEXT,__info_plist,/Users/Beingessner/dev/swift/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/internal/SwiftExperimental/Info.plist
>> -Wl,-application_extension
>> -L/Users/Beingessner/dev/swift/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./lib/swift/macosx/x86_64
>> -L/Users/Beingessner/dev/swift/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/../lib/swift/macosx/x86_64
>> -L/Users/Beingessner/dev/swift/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/./bin/../lib/swift/macosx
>> -o lib/swift/macosx/x86_64/libswiftSwiftExperimental.dylib -install_name
>> @rpath/libswiftSwiftExperimental.dylib
>> stdlib/internal/SwiftExperimental/macosx/x86_64/SwiftExperimental.o
>> -L/Users/Beingessner/dev/swift/build/Ninja-RelWithDebInfoAssert/llvm-macosx-x86_64/./lib
>> lib/swift/macosx/x86_64/libswiftCore.dylib -framework Foundation -framework CoreFoundation && :
>> Undefined symbols for architecture x86_64:
>>  "type metadata accessor for Swift.(VariantSetStorage in _69BD46AE5A11D8B70E9CADC1ECE9CF76)",
>> referenced from:
>>      Swift.Set.formUnion <A where A1: Swift.Sequence, A == A1.Iterator.Element> (A1) -> () in
>> SwiftExperimental.o
>>      Swift.Set.init <A where A1: Swift.Sequence, A == A1.Iterator.Element> (A1) -> Swift.Set<A> in
>> SwiftExperimental.o
>>      Swift.Set.intersection (Swift.Set<A>) -> Swift.Set<A> in SwiftExperimental.o
>>      Swift.Set.formSymmetricDifference (Swift.Set<A>) -> () in SwiftExperimental.o
>> ld: symbol(s) not found for architecture x86_64
>> clang: error: linker command failed with exit code 1 (use -v to see invocation)
>> [648/1027] Compiling
>> /Users/Beingessner/dev/swift/build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/stdlib/public/SwiftOnoneSupport/macosx/x86_64/SwiftOnoneSupport.o
>> ninja: build stopped: subcommand failed.
>> utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting
>> 
>>> On Nov 1, 2016, at 12:10 PM, Alexis via swift-dev <swift-dev at swift.org> wrote:
>>> 
>>> I made a PR with some basic privacy usage for Array, and everything seems to be working on OSX,
>> but not Linux.
>>> 
>>> https://github.com/apple/swift/pull/5573 <https://github.com/apple/swift/pull/5573> <https://github.com/apple/swift/pull/5573 <https://github.com/apple/swift/pull/5573>>
>>> 
>>> Exit Code: 1
>>> 
>>> Command Output (stderr):
>>> --
>>> /usr/bin/ld.gold: error:
>>> /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/libswiftSwiftOnoneSupport.a(SwiftOnoneSupport.o):
>>> multiple definition of
>>> '_TTSfq4n_d___TTSgq5Si___TZFVs15ContiguousArrayP33_4E8E8F8E53AAC7AA89B428DCA7DB222B11_copyBufferfRGVs22_ContiguousArrayBufferx_T_'
>>> /usr/bin/ld.gold:
>>> /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/libswiftCore.a(Swift.o):
>>> previous definition here
>>> /usr/bin/ld.gold: error:
>>> /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/libswiftSwiftOnoneSupport.a(SwiftOnoneSupport.o):
>>> multiple definition of
>>> '_TTSfq4n_d___TTSgq5Vs4Int8___TZFSaP33_4E8E8F8E53AAC7AA89B428DCA7DB222B11_copyBufferfRGVs22_ContiguousArrayBufferx_T_'
>>> /usr/bin/ld.gold:
>>> /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/libswiftCore.a(Swift.o):
>>> previous definition here
>>> /usr/bin/ld.gold: error:
>>> /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/libswiftSwiftOnoneSupport.a(SwiftOnoneSupport.o):
>>> multiple definition of
>>> '_TTSfq4n_d___TTSgq5Vs5UInt8___TZFSaP33_4E8E8F8E53AAC7AA89B428DCA7DB222B11_copyBufferfRGVs22_ContiguousArrayBufferx_T_'
>>> /usr/bin/ld.gold:
>>> /home/buildnode/jenkins/workspace/swift-PR-Linux/branch-master/buildbot_linux/swift-linux-x86_64/lib/swift_static/linux/libswiftCore.a(Swift.o):
>>> previous definition here
>>> clang-4.0: error: linker command failed with exit code 1 (use -v to see invocation)
>>> <unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
>>> 
>>> --
>>> 
>>> Close... (made copyBuffer private)
>>> 
>>> 
>>>> On Oct 29, 2016, at 6:04 PM, Dave Abrahams via swift-dev <swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>> wrote:
>>>> 
>>>> 
>>>> on Sat Oct 29 2016, Alexis <swift-dev-AT-swift.org <http://swift-dev-at-swift.org/> <http://swift-dev-at-swift.org/ <http://swift-dev-at-swift.org/>>> wrote:
>>>> 
>>>>>> On Oct 29, 2016, at 12:13 AM, Slava Pestov <spestov at apple.com <mailto:spestov at apple.com> <mailto:spestov at apple.com <mailto:spestov at apple.com>>> wrote:
>>>>>> 
>>>>>> It will become the default, but not yet, so yeah, you shouldn’t merge anything that only builds
>>>>> with this flag set.
>>>>>> 
>>>>>> Can you share the patch that adds private modifiers along with the
>>>>>> linker errors you are seeing? Now would be a good time to sort out
>>>>>> these issues.
>>>>> 
>>>>> I’m actually having trouble reproducing this now? I just rebased my
>>>>> branches onto master and using private/fileprivate on types, aliases,
>>>>> and functions seems to work perfectly fine (I tried a few things in
>>>>> Array and Dictionary). Did something interesting just get merged?
>>>> 
>>>> I suggest you try making and testing a pull request.  Often things seem
>>>> to work because of the way we build and test locally, but fail in the
>>>> full CI environment.
>>>> 
>>>>> 
>>>>>> 
>>>>>>> On Oct 28, 2016, at 4:16 PM, Alexis Beingessner <abeingessner at apple.com <mailto:abeingessner at apple.com> <mailto:abeingessner at apple.com <mailto:abeingessner at apple.com>>
>>>>> <mailto:abeingessner at apple.com <mailto:abeingessner at apple.com> <mailto:abeingessner at apple.com <mailto:abeingessner at apple.com>>>> wrote:
>>>>>>> 
>>>>>>> Won't merging anything relying on this flag break the build? Is this going to become the "new"
>>>>> default soon?
>>>>>>> 
>>>>>>> On Oct 28, 2016, at 6:43 PM, Slava Pestov <spestov at apple.com <mailto:spestov at apple.com> <mailto:spestov at apple.com <mailto:spestov at apple.com>> <mailto:spestov at apple.com <mailto:spestov at apple.com> <mailto:spestov at apple.com <mailto:spestov at apple.com>>>> wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Oct 23, 2016, at 4:13 PM, Michael Gottesman via swift-dev
>>>>>>>>> <swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>>> wrote:
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> On Oct 23, 2016, at 3:30 PM, Alexis Beingessner via swift-dev
>>>>>>>>>> <swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> Dave pointed out to me this week that the build crashes if the
>>>>>>>>>> stdlib tries to use private/fileprivate. I tried it myself and
>>>>>>>>>> lo and behold the linker can't find the private symbols. He
>>>>>>>>>> couldn't recall what about the build caused that, though.
>>>>>>>>>> 
>>>>>>>>>> Can anyone recall why this is? How hard is it to fix?
>>>>>>>>> 
>>>>>>>>> I am not 100% sure, but if it happens only with the stdlib and
>>>>>>>>> has to do with access control, I wouldn't be surprised if it has
>>>>>>>>> to do with -sil-serialize-all and friends. But I may be
>>>>>>>>> correct. I think Jordan is the right person to answer this
>>>>>>>>> question.
>>>>>>>>> 
>>>>>>>>> What do you think Jordan?
>>>>>>>>> Michael
>>>>>>>> 
>>>>>>>> Hi Alexis,
>>>>>>>> 
>>>>>>>> You can build the stdlib without sil-serialize-all now by passing a flag to build-script:
>>>>>>>> 
>>>>>>>> ./utils/build-script — --swift-stdlib-enable-resilience
>>>>>>>> 
>>>>>>>> Give that a shot and see if it fixes the issues you’re having with ‘private’.
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>>> swift-dev mailing list
>>>>>>>>>> swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>>
>>>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>>
>>>>> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>>>
>>>>>>>>> 
>>>>>>>>> _______________________________________________
>>>>>>>>> swift-dev mailing list
>>>>>>>>> swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>>
>>>>>>>>> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>>
>>>>> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>>>
>>>>>>>> 
>>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> swift-dev mailing list
>>>>> swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>
>>>>> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>>
>>>>> 
>>>> 
>>>> -- 
>>>> -Dave
>>>> 
>>>> _______________________________________________
>>>> swift-dev mailing list
>>>> swift-dev at swift.org <mailto:swift-dev at swift.org> <mailto:swift-dev at swift.org <mailto:swift-dev at swift.org>>
>>>> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>
>> <https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>>
>>> _______________________________________________
>>> swift-dev mailing list
>>> swift-dev at swift.org <mailto:swift-dev at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>
>> 
>> _______________________________________________
>> swift-dev mailing list
>> swift-dev at swift.org <mailto:swift-dev at swift.org>
>> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>
>> 
> 
> -- 
> -Dave
> 
> _______________________________________________
> swift-dev mailing list
> swift-dev at swift.org <mailto:swift-dev at swift.org>
> https://lists.swift.org/mailman/listinfo/swift-dev <https://lists.swift.org/mailman/listinfo/swift-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-dev/attachments/20161101/a2eae0de/attachment.html>

More information about the swift-dev mailing list