The current tokenizer is a modification of the ClassicTokenizer, both are constructed using jflex. They essentially split the incoming text into tokens as defined by whitespace and attempt to identify the type of the token (alphanumeric, numeric, Chinese, Control and Punctuation chars). Our version has some additional handling to do with acronums and apostrophes. Once the text has been tokenized then further adjustments can be based with filters.

There are a number of problems with our tokenizer
1. It doesnt handle non-european languages very well
2. It is difficult to modify because the code is generated with jflex which is both confusing and limited
3. Some of the custom code it includes could be simplified because of subsequent improvements in Lucene
4. It prevents us using a shipped tokenizer whihc would be much simpler.

So I looked at the new StandardTokenizer in Lucene 4.0 which covers languages such as Chinese, Thai ectera much better than ours because it recognizes all different types of wordboundaries not just whitespace according to this Unicode standard http://www.unicode.org/reports/tr29/ . But it has one massive problem for us it generally remove punctuation characters which it generally considers a word boundary char. But we need to keep our punctuation a because it is is key component of many artist names, in some cases it is is the only component (i.e !!!) , we do currently remove some punctuation in a filter applied after tokenization but not if that leaves that token empty because it only contains punctuation (i.e !!!). This new tokenizer is much more complex and I dont think it would be possible to amend it to keep the punctuation, it is certainly beyond my capabilities.

We have a very difficult problem to solve with a MusicBrainzAnalyzer (of which tokenizer is a key component). We want it to handle all languages without telling it what language it is handling, and we want to index text that is not proper words as well containing weird punctuation and symbols that break the rules.

So my plan is to create a new pure tokenizer use one of the simpler tokenizers (such as WhitespaceTokenizer) augmented with code from some of the specialist tokenizers for soem other scripts (i.e ThaiTokenizer, ChineseTokenizer) when we can identify the language. These tokenizers are pure Java removing the jflex issue from the equation making much easier to amend.