[swift-evolution] adding automated-testing of uncompilable features in XCTest

[Jeremy Pereira]

The Swift compiler can give you the information that you want. 

Here is what you can do:

Write a Swift source file that creates an instance of the type you want to test and then tries to access each of the private members. 

Write a shell script to compile this source file i a module with the file the type is defined in. Have it capture all the error messages by redirecting stderr and then count them. If it doesn’t have the right number, have the shell script emit a message that looks like a Swift error message.

Install the script in a run script build phase. Now you will get an error every time one of your private properties or methods loses its access modifier.

Personally, I wouldn’t bother. Any test to make sure that private members are private requires a separately maintained list of the private members. If somebody isn’t disciplined enough to add the word “private” to the beginning of a definition, they almost certainly aren’t going to bother updating a separate list. And the consequences of omitting “private” are only that the module has visibility of it. That’s not a huge deal. 


> On 13 Dec 2016, at 15:57, Benjamin Spratling via swift-evolution <swift-evolution at swift.org> wrote:
> 
> Howdy,
> 
> If my immediate goal were to verify private member values at intermediate steps in some process, which is the concept under consideration in the article you provided, I could access their values with a Mirror.
> 
> That's not my immediate goal.  My immediate goal is to falsify that the property is private.  The logic in the article asserts that a member is private, then asserts that because the member is private we do not need to test it.  But it fails to provide an automated test that the member is indeed private and remains so.  To accomplish that goal without an automated unit test is painstakingly tedious for a human.
> 
> And as to whether the logic in the article is correct, assuming its assumptions are true, it is still only one kind of testing.  Depending on the field it might be called "interface" or "black box" testing, and they rightly call that out.  But it would not be considered to be the only kind of testing which needs to be done.
> 
> -Ben
> 
> Sent from my iPhone.
> 
> On Dec 13, 2016, at 1:35 AM, Jean-Daniel <mailing at xenonium.com> wrote:
> 
>> An interesting reading about testing private members:
>> 
>> https://cocoacasts.com/how-to-unit-test-private-methods-in-swift/
>> 
>> 
>>> Le 12 déc. 2016 à 06:10, Derrick Ho via swift-evolution <swift-evolution at swift.org> a écrit :
>>> 
>>> It bugs me as well that we can not test private components using XCTest. Objective-c never had this problem since privacy doesn't exist. I would like to see a version of XCTest that allows us to test every component.
>>> 
>>> 
>>> On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <swift-evolution at swift.org> wrote:
>>> Howdy,
>>>         I’d like to see how much interest there is for adding these to the XCTest module.  If I have missed some pro or con, or missed a technical point, your feedback is welcome before I go to the lengths to draw up a formal proposal.
>>> 
>>> 
>>> There are several features of Swift which cannot be effectively automated-tested.
>>> For instance, when a type or one of its members is private or file private, an outside test suite cannot access it, and the information that the type is private is not included in a Mirror.  There are similar concerns for testability with internal, and public access levels.  Tests can be written ensuring these access levels are >= some level, but not == or < some level.  In other words the very usefulness of these features cannot be tested.
>>> 
>>> Other attributes to be tested in this way are:
>>> 
>>> - Mutability of a member.  No automated test can be written which verifies that a function cannot be called on a let struct, or that a property cannot be set at a specific access level.
>>> 
>>> - That a stored property is weak or unowned.
>>> 
>>> - That a class or class member is “final”
>>> 
>>> These are concepts which need to be unit-tested to ensure good design is not broken, but do not need to be included in a release build, so including them in the XCTest framework seems like an appropriate destination.
>>> Moreover, the information for all of these features exists in the .swiftmodule files, which are included in test builds, but sometimes stripped for release.
>>> 
>>> Examples:
>>> Since these features inherently have to do with testing features which cannot be stated in compiled code, I recommend specifying names with Strings.  Here are some examples of what I would like to write in my test code:
>>> 
>>> XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel, .private)
>>> 
>>> XCTAssertEqual( Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable, false)
>>> 
>>> 
>>> Alternatives:
>>> 1.      Building an independent .swiftmodule parser in a single Swift module, which can be included in test builds.
>>>                 + Can be distributed independently from Swift sources, requiring 0 buy-in from Swift community
>>>                 + requires a single additional module for the test.
>>>                 - Depends on ever-changing binary interface.
>>>                 : Intractable, not DRY
>>> 
>>> 2.      Use existing sourcekitd.
>>>                 + harnesses changes in the compiler’s source code with SourceKit
>>>                 - cannot be run in a test suite without extensive work by user to configure bundles explicitly.
>>>                         Exceptionally poor user experience
>>>                 : sourcekitd XPC architecture only works on macOS
>>> 
>>> 3.      Use a standalone tool for tests
>>>                 + harnesses changes in the compiler’s source code with SourceKit
>>>                 + no installation in user’s source code necessary
>>>                 : cannot be effectively run by SPM test infrastructure
>>> 
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org
>>> https://lists.swift.org/mailman/listinfo/swift-evolution
>> 
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution