[swift-evolution] adding automated-testing of uncompilable features in XCTest

[Brian Gesiak]

Hi Benjamin,

I think your proposal is interesting, and you're certainly right: XCTest
has no way to test accessibility levels, or the other attributes you
mention. However, I wonder, what do you hope to achieve by adding an
automated test for accessibility levels?

Maybe your goal is to verify the accessibility level is what you think it
is. However, in my experience, accessibility levels are fairly
straightforward: if I specify a variable is 'public', it's public. If I
specify it to be 'fileprivate', it's private to the file. I could write an
automated test to verify the accessibility level, but the test would
provide me with very little value.

Maybe your goal is to ensure that other programmers don't accidentally
change the accessibility levels. Again, I think because they're
straightforward, I don't think there's much danger there.

To me, tests for accessibility would be redundant. It would be like adding
a test to verify that my struct 'Foo' has a member named 'bar'. If I ever
decided to rename 'Foo.bar', I would have to update the test. But I would
never "accidentally" rename 'Foo.bar' -- if I changed the name, it's
because I modified the code for that purpose. A test adds overhead, but no
benefit.

I feel the same about the 'mutating', 'weak', 'unowned', and 'final'.

It's definitely subjective, though. You might feel that the tests I
describe above are valuable, and you're entitled to that opinion.

If you'd still like to pursue this idea, I think your Swift Evolution
proposal would benefit from research on how other languages and testing
frameworks approach this problem. How do C++ programmers test whether an
attribute is private? How do Java developers test that a class is final?
What about Rust or Go?

- Brian Gesiak


On Mon, Dec 12, 2016 at 12:10 AM, Derrick Ho via swift-evolution <
swift-evolution at swift.org> wrote:

> It bugs me as well that we can not test private components using XCTest.
> Objective-c never had this problem since privacy doesn't exist. I would
> like to see a version of XCTest that allows us to test every component.
>
>
>
> On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <
> swift-evolution at swift.org> wrote:
>
>> Howdy,
>>         I’d like to see how much interest there is for adding these to
>> the XCTest module.  If I have missed some pro or con, or missed a technical
>> point, your feedback is welcome before I go to the lengths to draw up a
>> formal proposal.
>>
>>
>> There are several features of Swift which cannot be effectively
>> automated-tested.
>> For instance, when a type or one of its members is private or file
>> private, an outside test suite cannot access it, and the information that
>> the type is private is not included in a Mirror.  There are similar
>> concerns for testability with internal, and public access levels.  Tests
>> can be written ensuring these access levels are >= some level, but not ==
>> or < some level.  In other words the very usefulness of these features
>> cannot be tested.
>>
>> Other attributes to be tested in this way are:
>>
>> - Mutability of a member.  No automated test can be written which
>> verifies that a function cannot be called on a let struct, or that a
>> property cannot be set at a specific access level.
>>
>> - That a stored property is weak or unowned.
>>
>> - That a class or class member is “final”
>>
>> These are concepts which need to be unit-tested to ensure good design is
>> not broken, but do not need to be included in a release build, so including
>> them in the XCTest framework seems like an appropriate destination.
>> Moreover, the information for all of these features exists in the
>> .swiftmodule files, which are included in test builds, but sometimes
>> stripped for release.
>>
>> Examples:
>> Since these features inherently have to do with testing features which
>> cannot be stated in compiled code, I recommend specifying names with
>> Strings.  Here are some examples of what I would like to write in my test
>> code:
>>
>> XCTAssertEqual( Module(named:”SingMusicLayout”
>> )?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel,
>> .private)
>>
>> XCTAssertEqual( Module(named:”SingMusicLayout”
>> )?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(
>> for:PitchInterval)->CGFloat”)?.mutable, false)
>>
>>
>> Alternatives:
>> 1.      Building an independent .swiftmodule parser in a single Swift
>> module, which can be included in test builds.
>>                 + Can be distributed independently from Swift sources,
>> requiring 0 buy-in from Swift community
>>                 + requires a single additional module for the test.
>>                 - Depends on ever-changing binary interface.
>>                 : Intractable, not DRY
>>
>> 2.      Use existing sourcekitd.
>>                 + harnesses changes in the compiler’s source code with
>> SourceKit
>>                 - cannot be run in a test suite without extensive work by
>> user to configure bundles explicitly.
>>                         Exceptionally poor user experience
>>                 : sourcekitd XPC architecture only works on macOS
>>
>> 3.      Use a standalone tool for tests
>>                 + harnesses changes in the compiler’s source code with
>> SourceKit
>>                 + no installation in user’s source code necessary
>>                 : cannot be effectively run by SPM test infrastructure
>>
>> _______________________________________________
>> swift-evolution mailing list
>> swift-evolution at swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
>>
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20161212/c7a94142/attachment.html>