Unicode segmenter performance

[clipperhouse]

In this PR, as an experiment I’ve swapped out the Unicode segmenter in Bleve with this one.

Benefits

• ~2x throughput improvement
• ~300x lower allocation

Previous
BenchmarkTokenizeMultilingual-8   	     500	   2181603 ns/op	  49.95 MB/s	     16042 tokens	 1374693 B/op	     610 allocs/op

New
BenchmarkTokenizeMultilingual-8   	     932	   1087282 ns/op	 100.23 MB/s	     16042 tokens	 1310723 B/op	       2 allocs/op

(Run on my M2 MacBook.)

• Updates Unicode 8 → Unicode 15

Testing & compatibility

• Both segmenters pass the official Unicode test suites
• Bleve tests pass
• Wrote a test to demonstrate identical results from both segmenters on multilingual sample text
• Added adaptors to the underlying UAX29 package to determine Bleve token types
• Added fuzzing to the UAX29 package
• Known differences:
  • The original segementer splits runs of spaces into separate tokens; UAX29 concatenates runs of spaces into a single token. This should be irrelevant since Bleve filters whitespace in any case.
  • The original segmenter doesn’t handle emoji skin tone modifiers, the new one does. I attribute this to the newer Unicode version, not a bug.

[abhinavdangeti]

Thank you for submitting this @clipperhouse , we'll review it soon.

[clipperhouse]

Thanks @abhinavdangeti. I did some more testing of differences between segmenters here: https://github.com/clipperhouse/segmenter-repro

[clipperhouse]

Updated benchmark, using new sample text (~110K in size, multilingual). Note allocations.

Previous
BenchmarkTokenizeEnglishText-4   	     285	   4167394 ns/op	  26.15 MB/s	     16042 tokens	 1348515 B/op	     610 allocs/op

New
BenchmarkTokenizeEnglishText-4   	     614	   1986500 ns/op	  54.86 MB/s	     16042 tokens	 1310730 B/op	       2 allocs/op

[clipperhouse]

@abhinavdangeti I think this is in good shape for your review. Happy to discuss.

[clipperhouse]

I see that the go:build syntax isn’t compatible with old Go versions, fix incoming.

[clipperhouse]

@abhinavdangeti OK, try those workflows again? Thanks.

[clipperhouse]

I rebased a bit for a cleaner merge.

[clipperhouse]

(Rebased)

[clipperhouse]

@abhinavdangeti Friendly ping, let me know if you'd like to pursue this.

[clipperhouse]

Hiya @abhinavdangeti, the above pushes are just rebases, no updates here in a while. Would you like to run checks?

[abhinavdangeti]

Thanks @clipperhouse , re-running the checks.

[clipperhouse]

Looks good, thanks. Ready for review at your convenience.