<font face="Verdana,Arial,Helvetica,sans-serif" size="2"><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">Hi Philippe, other interested people:</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">Here is the detailed description of an apparent bug in NSRegularExpression along with a possible solution. Request you to comment on the solution, in the context of correctness and performance. </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"> </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">This is long - please bear with me! Alternatively, we could discuss this over a pull request if you concur.</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">---> </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">The following test case has been borrowed from some of those tests in TestNSRegularExpression that aren’t exercised (test_complexRegularExpression) as of today:</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">import Foundation</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier; min-height: 17px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">let searchStr = "This this is the theway."</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">let testRegex = try NSRegularExpression(pattern: "\\b(th[a-z]+) \\1\\b", options: [])</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">let fm = testRegex.firstMatchInString(searchStr, options: .WithTransparentBounds, range: NSMakeRange(0,20))</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">let str = NSString.init(string: searchStr)</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">if let match = fm {</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> print("Test failed")</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> print(str.substringWithRange(match.range))</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">} else {</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> print("Test passed")</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">}</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">The test fails on Linux - a false match is reported. The substring “the the” matches pattern “\b(th[a-z]+) \1\b” which is wrong because the second “the” does not occur on a word boundary. Note that we are using the option: WithTransparentBounds. This means the matcher can look beyond the search range, for word boundaries. </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">The question is why the word boundary metacharacter at the end of the patter isn’t being honoured. I studied the functions _CFRegularExpressionEnumerateMatchesInString() and prepareRegularExpression() from CFRegularExpression.c and these are my findings that provide an explanation for a possible reason:</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">1. We first try to get the search text - a UniChar* - using CFStringGetCharactersPtr(). I guess this is done to try improve performance. We copy the entire search text and set regionStart and regionLimit to the search range.</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">2. Alternatively, if CFStringGetCharactersPtr() fails, we try to fill the UniChar buffer using CFStringGetCharacters(). Here we try to reduce the size of the search text so that it matches the search range. We use an “enclosingRange” for this. For “transparentBounds” we use the entire text. For nonAnchoringBounds we just take the searchRange plus one character to the left (to match ^) plus one to the right (to match $) : <span style="font-family: Courier;">enclosingRange = range; </span><span style="font-family: Courier;"> </span></font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">if (transparentBounds) {</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> enclosingRange = CFRangeMake(0, length);</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">} else if (nonAnchoringBounds) {</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> if (enclosingRange.location > 0) {</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> enclosingRange.location--;</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> enclosingRange.length++;</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> }</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> if (enclosingRange.location + enclosingRange.length < length) enclosingRange.length++;</font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545">}</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">We then set regionStart and regionLimit to the search range.</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">3. Further we set the search text using the ICU function uregex_setText() like this: </font></p><p style="font-size: 14px; line-height: normal; font-family: Courier;"><font color="#454545"> uregex_setText(regex, (const UChar *)stringBuffer, (int32_t)regionLimit, &errorCode);</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"> </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">Note that we use “regionLimit” for the “textLength”, which seems questionable. This truncates the search text down to a substring matching the specified search range. So, in the above case where the search text is : "<b>This this is the theway.</b>" , using a search range of {0,20} the search text that we actually pass to ICU is “<b>This this is the the</b>” which matches “\b(th[a-z]+) \1\b”. Though this may not be a problem in most searches, the <b>WithTransparentBounds</b> option fails to take effect. </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"><br></font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">Proposed solution</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">————————</font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545">The straightforward solution is to simply set the length of the search text to the actual length in <font color="#000000">prepareRegularExpression()</font> of the search string as follows: </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><font color="#454545"> </font></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"><font color="#454545"> </font><b>int32_t textLength = length; //proposed fix</b></p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><br></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> …</p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><br></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (range.location + range.length <= INT_MAX) stringBuffer = (UniChar *)CFStringGetCharactersPtr(string);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (stringBuffer) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> regionStart = (int64_t)range.location;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> regionLimit = (int64_t)(range.location + range.length);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> *offset = 0;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> } else {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> enclosingRange = range;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (transparentBounds) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> enclosingRange = CFRangeMake(0, length);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> } else if (nonAnchoringBounds) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (enclosingRange.location > 0) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> enclosingRange.location--;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> enclosingRange.length++;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (enclosingRange.location + enclosingRange.length < length) enclosingRange.length++;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><br></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> …</p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><br></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> regionStart = (int64_t)(range.location - enclosingRange.location);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> regionLimit = (int64_t)((range.location + range.length) - enclosingRange.location);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> *offset = enclosingRange.location;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (enclosingRange.length <= STACK_BUFFER_SIZE) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> stringBuffer = stackBuffer;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (enclosingRange.length > 0) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> CFStringGetCharacters(string, enclosingRange, stringBuffer);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> } else {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> stringBuffer = (UniChar *)malloc(sizeof(UniChar) * enclosingRange.length);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (stringBuffer) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> CFStringGetCharacters(string, enclosingRange, stringBuffer);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> *bufferToFree = stringBuffer;</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> <b>textLength = enclosingRange.length; //proposed fix</b></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><p style="font-size: 14px; line-height: normal; font-family: Arial; min-height: 16px;"><br></p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> if (stringBuffer) {</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> regex = checkOutRegularExpression(internal, checkout, checkedOutRegex);</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> uregex_setText(regex, (const UChar *)stringBuffer, <b>textLength</b>, &errorCode); //proposed fix</p><p style="font-size: 14px; line-height: normal; font-family: Arial;"> }</p><div><br></div><font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div class="socmaildefaultfont" dir="ltr" style="font-family:Arial;font-size:10.5pt"><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div dir="ltr">Thanks!</div><div dir="ltr"><br></div><div dir="ltr"><br><span style="font-family:georgia,serif;"><span style="font-size: 1.143em;">Pushkar N Kulkarni,</span></span></div>
<div dir="ltr"><span style="font-family:georgia,serif;"><span style="font-size: 1.143em;">IBM Runtimes</span></span></div>
<div dir="ltr"> </div>
<div dir="ltr"><em><span style="font-family:georgia,serif;"><span style="font-size: 0.857em;">"Any sufficiently advanced technology is indistinguishable from magic." - Arthur Clarke</span></span></em></div></div></div></div></font></font><BR>