[swift-evolution] adding automated-testing of uncompilable features in XCTest

[Derrick Ho]

It bugs me as well that we can not test private components using XCTest.
Objective-c never had this problem since privacy doesn't exist. I would
like to see a version of XCTest that allows us to test every component.


On Mon, Dec 12, 2016 at 12:02 AM Benjamin Spratling via swift-evolution <
swift-evolution at swift.org> wrote:

> Howdy,
>         I’d like to see how much interest there is for adding these to the
> XCTest module.  If I have missed some pro or con, or missed a technical
> point, your feedback is welcome before I go to the lengths to draw up a
> formal proposal.
>
>
> There are several features of Swift which cannot be effectively
> automated-tested.
> For instance, when a type or one of its members is private or file
> private, an outside test suite cannot access it, and the information that
> the type is private is not included in a Mirror.  There are similar
> concerns for testability with internal, and public access levels.  Tests
> can be written ensuring these access levels are >= some level, but not ==
> or < some level.  In other words the very usefulness of these features
> cannot be tested.
>
> Other attributes to be tested in this way are:
>
> - Mutability of a member.  No automated test can be written which verifies
> that a function cannot be called on a let struct, or that a property cannot
> be set at a specific access level.
>
> - That a stored property is weak or unowned.
>
> - That a class or class member is “final”
>
> These are concepts which need to be unit-tested to ensure good design is
> not broken, but do not need to be included in a release build, so including
> them in the XCTest framework seems like an appropriate destination.
> Moreover, the information for all of these features exists in the
> .swiftmodule files, which are included in test builds, but sometimes
> stripped for release.
>
> Examples:
> Since these features inherently have to do with testing features which
> cannot be stated in compiled code, I recommend specifying names with
> Strings.  Here are some examples of what I would like to write in my test
> code:
>
> XCTAssertEqual(
> Module(named:”SingMusicLayout”)?.type(named:”NoteSetter”)?.property(named:”session”)?.accessLevel,
> .private)
>
> XCTAssertEqual(
> Module(named:”SingMusicLayout”)?.type(named:”ScaleLayout”)?.method(named:”baselineOffset(for:PitchInterval)->CGFloat”)?.mutable,
> false)
>
>
> Alternatives:
> 1.      Building an independent .swiftmodule parser in a single Swift
> module, which can be included in test builds.
>                 + Can be distributed independently from Swift sources,
> requiring 0 buy-in from Swift community
>                 + requires a single additional module for the test.
>                 - Depends on ever-changing binary interface.
>                 : Intractable, not DRY
>
> 2.      Use existing sourcekitd.
>                 + harnesses changes in the compiler’s source code with
> SourceKit
>                 - cannot be run in a test suite without extensive work by
> user to configure bundles explicitly.
>                         Exceptionally poor user experience
>                 : sourcekitd XPC architecture only works on macOS
>
> 3.      Use a standalone tool for tests
>                 + harnesses changes in the compiler’s source code with
> SourceKit
>                 + no installation in user’s source code necessary
>                 : cannot be effectively run by SPM test infrastructure
>
> _______________________________________________
> swift-evolution mailing list
> swift-evolution at swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20161212/44f8898e/attachment.html>