[swift-evolution] String update
John Holdsworth
mac at johnholdsworth.com
Mon Jan 15 18:03:11 CST 2018
> On 12 Jan 2018, at 02:01, Michael Ilseman <milseman at apple.com> wrote:
>
>
>
>> On Jan 11, 2018, at 9:49 AM, John Holdsworth <mac at johnholdsworth.com <mailto:mac at johnholdsworth.com>> wrote:
>>
>> Hi Michael,
Thanks for getting back
>> Thanks for sending this through. It’s an interesting read. One section gave me pause however. I feel Swift should resist the siren call of combining Swift Syntax with Regex syntax as it falls on the wrong side of Occam's razor. ISO Regex syntax is plenty complex enough without trying to incorporate named capture, desirable as it may be. Also, if I was to go down that route, I’d move away from / as the delimiter which is a carry over from Perl to something like e”I am a regex” to give the lexer more to go on which could represent say, a cached instance of NSRegularExpression.
>>
>
> Sorry for the confusion, in no way is the syntax of regex literals tied to any syntactic standard or historical baggage. It might happen to align when obvious or beneficial to common practice, i.e. use same basic meta-characters and built in character classes. This is important as they certainly wouldn’t honor any standard semantically, other than perhaps UTS-18 level-2 by coincidence (which doesn’t dictate preference of ambiguous matches, AFAIK).
>
> As a downside, this does open a huge domain of bike shedding ;-)
>
> The approach mentioned would allow someone (e.g. SPM packages) to provide functionality such as (ignoring style) for execution on Swift’s regex engine:
>
> func compilePOSIX(_: String) throws -> Regex<[Any]> // Or perhaps Regex<[Substring]>, or Regex<POSIXMatch>, details...
> func compileRE2(_: String) -> Regex<[Any]> // ditto
> … PCRE, ICU, JS, Perl 5, Perl 6, etc. ...
>
> (Note that we can't use NSRegularExpression as an execution engine out-of-the-box, as it relies on ICU which doesn’t provide matching modulo canonical equivalence. Not to mention the performance issues….)
I see you’re prepared to really lift the lid on what is a regex literal !
I've read your comments and completely rewritten the regex playground using a a generic regex object with generic subscripts and the result was definitely a case of less code is more. It also neatly separates the problem out into two parts. The first: conversion from the new compiler supported literal to a typed generic and the second: what is possible using the subscript notation implemented today using string regex literals. What you loose using strings for the pattern literal is the type inference and being able to name captures in the literal. See the new playground for details.
>> And now for something completely different...
>>
>> Common usage patterns for a regex fall into 4 categories: deconstruction, replacement, iteration and switch/case.
>
> Could you elaborate more on this breakdown? What are the differences between deconstruction, iteration, and switch/case?
I’m thinking of the common regex operations
detect a match
extract the first match
extract the group captures of the first match
extract all matches
extract the group captures of all matches
assign to all of the above
iterate lazily over match/group captures.
pass closure over all matches.
& Is there something interesting to be done for case statements?
>> Ideally the representation of a regex match would the same for all four of these categories and I’d like to argue a set of expressive regex primitives can be created without building them into the language.
>>
>
> BTW, the “built into the language” would be confined to the regex literal syntax. The Regex<T> type wouldn’t necessarily need to be built-in, and could be constructed through other means.
>
>> I’ve talked before about a regex match being coded as a string/regex subscripting into a string and I’ve been able to move this forward since last year. While this seems like an arbitrary operator to use it has some semantic sense in that you are addressing a sub-part of the string with pattern as you might use an index or a key. Subscripts also have some very interesting properties in Swift compared to other operators or functions: You don’t have to worry about precedence, they can be assigned to, used as an interator, and I've learned since my last email on this topic that the Swift type checker will disambiguate multiple subscript overloads on the basis of the type of the variable is being assigned to.
>>
>
> Why do you use String as a regex rather than a new type, which could be ExpressibleByStringLiteral? That might help with overloading or ambiguities, and a new type is a something we can extend with regex-specific functionality.
>
> String could have a generic subscript from Regex<T> to T. Perhaps it could also be done as a setter, assigning a value of T (which may have to be string convertible... details).
I’ve used a protocol for string regex literals so they can include regex options i.e. "(\\w)(\\w*)".caseInsensitive so the gap between string as a literal and compiler supported version is less.
public protocol RegexLiteral {
var regexPattern: String { get }
var regexOptions: NSRegularExpression.Options { get }
}
>> An extension to String can now realise the common use cases by judicious use of types:
>>
>> var input = "Now is the time for all good men to come to the aid of the party"
>>
>> if input["\\w+"] {
>> print("match")
>> }
>>
>> // receiving type controls data you get
>> if let firstMatch: Substring = input["\\w+"] {
>> print("match: \(firstMatch)")
>> }
>>
>> if let groupsOfFirstMatch: [Substring?] = input["(all) (\\w+)"] {
>> print("groups: \(groupsOfFirstMatch)")
>> }
>>
>> // "splat" out up to N groups of first match
>> if let (group1, group2): (String, String) = input["(all) (\\w+)"] {
>> print("group1: \(group1), group2: \(group2)")
>> }
>>
>> if let allGroupsOfAllMatches: [[Substring?]] = input["(\\w)(\\w*)"] {
>> print("allGroups: \(allGroupsOfAllMatches)")
>> }
>>
>
> I’m interested in how you view the tradeoffs of not introducing a new type. If there was a Regex<T>, it could have computed properties for, e.g. an eager allMatches, lazy allMatches, firstMatch (given some ordering semantics), ignoringCaptures, caseInsensitive, …. then you don’t need your “(all) ” directives.
“all” directives in the new version come out of the type context you assign the match to/replace from. If it is an array then it is “all” otherwise first match operations.
if let match: (String, String, String) = numbers["(\\d+) (\\d+)-(\\d+)"] {
print(match)
}
numbers["(\\d+) (\\d+)-(\\d+)"] = ("555", "777", "1234")
XCTAssertEqual(numbers, "phone: 555 777-1234 fax: 555 666-4321")
// arrays of tuples operate on all matches
let matches2: [(String, String, String)] = numbers["(\\d+) (\\d+)-(\\d+)"]
print(matches2)
numbers["(\\d+) (\\d+)-(\\d+)"] = [("555", "888", "1234"), ("555", "999", "4321")]
XCTAssertEqual(numbers, "phone: 555 888-1234 fax: 555 999-4321")
>> // regex replace by assignment
>> input["men"] = "folk"
>> print(input)
>>
>
> Ok, you’re starting to sell me on the subscript-setter that takes a Regex ;-). The setter value wouldn’t be able to use information from the captures, so we would probably still want a substitute API that takes a closure receiving captures, but this looks nice for simple usage.
There's a public api on the underlying regex object for all operations which the subscripts use. In the case of a closure this can also be assigned to a match instead of a template though this borders on obfuscation:
str["(\\w)(\\w*)"] = {
(groups: (first: String, rest: String), stop) -> String in
return groups.first+groups.rest.uppercased()
}
which subscripts translate to:
str = Regex<(String, String)>(pattern: "(\\w)(\\w*)").replacing(target: str, exec: {
(groups: (first: String, rest: String), stop) -> String in
return groups.first+groups.rest.uppercased()
})
> Transcribing into the presented approach (Using ｢ and ｣ as delimiters in a surely-futile effort to not focus on specific syntax):
>
> input[｢\d+｣.firstMatch] = 123
> input[｢\d+｣.allMatches] = sequence(first: 42) { return $0 + 1 }
This is how you’d have to do it if you relied on the literal to contain the type.
>> // parsing a properties file using regex as iterator
>> let props = """
>> name1 = value1
>> name2 = value2
>> """
>>
>> var params = [String: String]()
>> for groups in props["(\\w+)\\s*=\\s*(.*)"] {
>> params[String(groups[1]!)] = String(groups[2]!)
>> }
>> print(params)
>
> Translating this over to the literal style:
>
> for (name, value) in props[｢(let _ = \w+) \s* = \s* (let _ = .*)｣.lineByLine] {
> print(name, value)
> }
.lineByLine is .regexLazy which changes the type of the literal to force an iterator.
> Or even better, give it a name!
>
> let propertyPattern = ｢(let name = \w+) \s* = \s* (let value = .*) // Regex<(name: Substring, value: Substring)>
> for (name, value) in props[propertyPattern.lineByLine] {
> print(name, value)
> }
This is an interesting case of where I wonder what the gain in naming the captures really is considering the disruption to regex syntax this would involve. The names in the assignment override them in the finish.
>> The case for switches is slightly more opaque in order to avoid executing the match twice but viable.
>>
>> let match = RegexMatch()
>> switch input {
>> case RegexPattern("(\\w)(\\w*)", capture: match):
>> let (first, rest) = input[match]
>> print("\(first) \(rest)")
>> default:
>> break
>> }
>>
>
> Using the literal approach:
>
> let peelFirstWordChar = ｢(let leading = \w)(let trailing = \w+)｣ // Regex<(leading: Substring, trailing: Substring)>, or perhaps Regex<(leading: Character, trailing: Substring>), details….
> switch input {
> case let (first, rest) <- peelFirstWordChar:
> print(“\(first) \(rest)”)
> }
For switch, in the absence of compiler support for bindings I came up with:
let match = RegexMatch()
switch str {
case "(\\w)(\\w*)".regex(capture: match):
let (first, rest): (String, String) = str[match]
print("\(first)~\(rest)")
default:
break
}
>> This is explored in the attached playground (repo: https://github.com/johnno1962/SwiftRegex4 <https://github.com/johnno1962/SwiftRegex4>)
>> <SwiftRegex4.playground.zip>
>>
>> I’m not sure I really expect this to take off as an idea but I’d like to make sure it's out there as an option and it certainly qualifies as “out there”.
>>
>
> I think it’s very interesting! Thanks for sharing. Do you have more usage examples?
I’ve expanded out the playground to cover most of the things you can do. The generic subscripts made the difference (Thanks Swift4!)
>> John
>>
>>> On 10 Jan 2018, at 19:58, Michael Ilseman via swift-evolution <swift-evolution at swift.org <mailto:swift-evolution at swift.org>> wrote:
>>>
>>> Hello, I just sent an email to swift-dev titled "State of String: ABI, Performance, Ergonomics, and You!” at https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20180108/006407.html <https://lists.swift.org/pipermail/swift-dev/Week-of-Mon-20180108/006407.html>, whose gist can be found at https://gist.github.com/milseman/bb39ef7f170641ae52c13600a512782f <https://gist.github.com/milseman/bb39ef7f170641ae52c13600a512782f>. I posted to swift-dev as much of the content is from an implementation perspective, but it also addresses many areas of potential evolution. Please refer to that email for details; here’s the recap from it:
>>>
>>> ### Recap: Potential Additions for Swift 5
>>>
>>> * Some form of unmanaged or unsafe Strings, and corresponding APIs
>>> * Exposing performance flags, and some way to request a scan to populate them
>>> * API gaps
>>> * Character and UnicodeScalar properties, such as isNewline
>>> * Generalizing, and optimizing, String interpolation
>>> * Regex literals, Regex type, and generalized pattern match destructuring
>>> * Substitution APIs, in conjunction with Regexes.
>>>
>>> _______________________________________________
>>> swift-evolution mailing list
>>> swift-evolution at swift.org <mailto:swift-evolution at swift.org>
>>> https://lists.swift.org/mailman/listinfo/swift-evolution <https://lists.swift.org/mailman/listinfo/swift-evolution>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20180116/5d1355d6/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SwiftRegex5.playground.zip
Type: application/zip
Size: 30190 bytes
Desc: not available
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20180116/5d1355d6/attachment.zip>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.swift.org/pipermail/swift-evolution/attachments/20180116/5d1355d6/attachment-0001.html>
More information about the swift-evolution
mailing list