regex to split a document into words

Say you wanted to split a document into words, but works like are'nt shouldnt be split on the ' and numbers like 10,004,333 should remain intact, but other punctuation should be removed from the resulting word array.

A good way to do this is using Scanner's findWithinHorizon and a regular expression. This way you dont need to read the entire document into memory before processing it.

<br />Pattern p = Pattern.compile("[A-Za-z]+(?:['-][A-Za-z]+)?|\\d+(?:[,.]\\d+)?");<br />Scanner sc = new Scanner(new File("/path/to/the/file.txt"));<br />String word = null;<br />while ((word = sc.findWithinHorizon(p, 0)) != null)<br />{<br /> // etc.<br />}<br />