The Regex & StringTokenizer Challenge!

Sometimes programming is difficult - You have to make large architectural decisions, with overreaching implications on maintainability, performance and security. Other times, programming is simply frustrating - Something 'simple' doesn't work the way you expected, you're seriously testing your logic and finally you bring out the paper and pen for some good-old pseudo code!

The problem...

I was doing some pro-bono processing work on an excel file, removing duplicate rows based on some complicated criteria. Preferring to keep my sanity I abandoned VBA so I exported the spreadsheet to CSV. The crunch is, some of the cells were blank and this produced unpredictable results in my code. Here's a sample of the data:

view plaincopy to clipboardprint?

1. ,,dsafdf,,15,,,,
2. ,,fdsfjladsjf,,13,,,,
3. df,,sdff,,bemail,1,1,,
4. dsf,,sffdsf,,bsgemail,1,1,,
5. token1,token2,token3,token4,token5,token6,token7,token8,token9

,,dsafdf,,15,,,,<br />,,fdsfjladsjf,,13,,,,<br />df,,sdff,,bemail,1,1,,<br />dsf,,sffdsf,,bsgemail,1,1,,<br />token1,token2,token3,token4,token5,token6,token7,token8,token9

I've added the fifth row for demonstration of the number of tokens. In a row that is complete, we are expecting 9 tokens. Their position is important!

Source Code

I've included some source code showing 3 different approaches:

view plaincopy to clipboardprint?

1. import java.util.StringTokenizer;
2. 
   
3. 
   
4. public class Processor {
5. 
   
6. public static void main (String[] args) {
7. 
   
8. final String _DELIM = ",";
9. 
   
10. //An array of the strings
11. String[] sourceStrings = {
12. ",,dsafdf,,15,,,,",
13. ",,fdsfjladsjf,,13,,,,",
14. "df,,sdff,,bemail,1,1,,",
15. "dsf,,sffdsf,,bsgemail,1,1,,",
16. "token1,token2,token3,token4,token5,token6,token7,token8,token9"
17. };
18. 
    
19. System.out.println("Approach One: String.split()");
20. for(String s: sourceStrings)
21. {
22. //do the processing on the strings using
23. //the delimiter as a regular expression;
24. String[] split = s.split(_DELIM);
25. 
    
26. System.out.println("Number of tokens : " + split.length);
27. }
28. 
    
29. System.out.println("Approach Two: StringTokenizer");
30. for(String s: sourceStrings)
31. {
32. StringTokenizer st = new StringTokenizer(s, _DELIM, false);
33. 
    
34. System.out.println("Number of tokens : " + st.countTokens());
35. }
36. 
    
37. System.out.println("\r\nApproach Three: StringTokenizer returning delimiter - Take One");
38. for(String s: sourceStrings)
39. {
40. 
    
41. StringTokenizer st = new StringTokenizer(s, _DELIM, true);
42. 
    
43. System.out.println("Number of tokens : " + st.countTokens());
44. }
45. }
46. }

import java.util.StringTokenizer;<br /><br /><br />public class Processor {<br /><br /> public static void main (String[] args) {<br /><br /> final String _DELIM = ",";<br /> <br /> //An array of the strings<br /> String[] sourceStrings = {<br /> ",,dsafdf,,15,,,,",<br /> ",,fdsfjladsjf,,13,,,,",<br /> "df,,sdff,,bemail,1,1,,",<br /> "dsf,,sffdsf,,bsgemail,1,1,,",<br /> "token1,token2,token3,token4,token5,token6,token7,token8,token9"<br /> };<br /> <br /> System.out.println("Approach One: String.split()");<br /> for(String s: sourceStrings)<br /> {<br /> //do the processing on the strings using<br /> //the delimiter as a regular expression; <br /> String[] split = s.split(_DELIM);<br /> <br /> System.out.println("Number of tokens : " + split.length);<br /> }<br /> <br /> System.out.println("Approach Two: StringTokenizer");<br /> for(String s: sourceStrings)<br /> {<br /> StringTokenizer st = new StringTokenizer(s, _DELIM, false);<br /> <br /> System.out.println("Number of tokens : " + st.countTokens());<br /> }<br /> <br /> System.out.println("\r\nApproach Three: StringTokenizer returning delimiter - Take One");<br /> for(String s: sourceStrings)<br /> {<br /> <br /> StringTokenizer st = new StringTokenizer(s, _DELIM, true);<br /> <br /> System.out.println("Number of tokens : " + st.countTokens());<br /> }<br /> }<br />}<br />

Output
A listing of the output of the code above is shown below:

view plaincopy to clipboardprint?

1. Approach One: String.split()
2. Number of tokens : 5
3. Number of tokens : 5
4. Number of tokens : 7
5. Number of tokens : 7
6. Number of tokens : 9
7. 
   
8. Approach Two: StringTokenizer
9. Number of tokens : 2
10. Number of tokens : 2
11. Number of tokens : 5
12. Number of tokens : 5
13. Number of tokens : 9
14. 
    
15. Approach Three: StringTokenizer returning delimiter - Take One
16. Number of tokens : 10
17. Number of tokens : 10
18. Number of tokens : 13
19. Number of tokens : 13
20. Number of tokens : 17

Approach One: String.split()<br />Number of tokens : 5<br />Number of tokens : 5<br />Number of tokens : 7<br />Number of tokens : 7<br />Number of tokens : 9<br /><br />Approach Two: StringTokenizer<br />Number of tokens : 2<br />Number of tokens : 2<br />Number of tokens : 5<br />Number of tokens : 5<br />Number of tokens : 9<br /><br />Approach Three: StringTokenizer returning delimiter - Take One<br />Number of tokens : 10<br />Number of tokens : 10<br />Number of tokens : 13<br />Number of tokens : 13<br />Number of tokens : 17<br />

Approach One: String.split()
The first approach uses the String.split() method passing in the delimiter as a regular expression. As can be seen, this fails the test as we have ranges of 5-9 tokens. The correct answer should be 9.

Approach Two: StringTokenizer
The second approach uses the StringTokenizer to split up the strings. Again, we get unpredictable results. 2 tokens for the first two lines, as it picks only the items that are not the delimiter. The Correct answer here would be 9 and for the items that are null, we should get a null token.

Approach Three: StringTokenizer returning the delimiter
The third approach uses the StringTokenizer with a different constructor that also returns the delimiter. Again, lots of wrong results. The correct number of tokens should be 17 - 8 commas and 9 non comma items, preferrably null.

So there you have it, a rather simple task (I would have thought), but I did bang my head against the wall trying to find the right solution. If I ever get to be a teacher, this would be great for an exam! There's the challenge! I'll post the solution soon.

UPDATE!

view plaincopy to clipboardprint?

1. public class NonCollapsingStringTokenizer {  
2.   
3.     private String str;  
4.     private String delim;  
5.     private int currentPosition;  
6.   
7.     private NonCollapsingStringTokenizer() {  
8.           
9.     }  
10.   
11.     public NonCollapsingStringTokenizer(String str, String delimStr) {  
12.         this.str = str;  
13.         this.delim = delimStr;  
14.     }  
15.   
16.     public String nextToken() {  
17.         int nextDelimPosition = str.length();  
18.         int delimPosition = str.indexOf(delim, currentPosition);  
19.         if (delimPosition >= 0 && delimPosition < nextDelimPosition) {  
20.             nextDelimPosition = delimPosition;  
21.         }  
22.         String token = str.substring(currentPosition, nextDelimPosition);  
23.         currentPosition = nextDelimPosition + 1;  
24.         return token;  
25.     }  
26.   
27.     public boolean hasMoreTokens() {  
28.         return (currentPosition < str.length());  
29.     }  
30.       
31.     public int countTokens() {  
32.         int count = 0;  
33.           
34.         NonCollapsingStringTokenizer clone = (NonCollapsingStringTokenizer) this.clone();  
35.         while(clone.hasMoreTokens())  
36.         {  
37.             clone.nextToken();  
38.             count++;  
39.         }  
40.           
41.         return count;  
42.     }  
43.       
44.     public Object clone() {  
45.         NonCollapsingStringTokenizer copy = new NonCollapsingStringTokenizer();  
46.         copy.str = str;  
47.         copy.delim = delim;  
48.         return currentPosition;  
49.           
50.     }  
51. }  

public class NonCollapsingStringTokenizer {<br /><br /> private String str;<br /> private String delim;<br /> private int currentPosition;<br /><br /> private NonCollapsingStringTokenizer() {<br /> <br /> }<br /><br /> public NonCollapsingStringTokenizer(String str, String delimStr) {<br /> this.str = str;<br /> this.delim = delimStr;<br /> }<br /><br /> public String nextToken() {<br /> int nextDelimPosition = str.length();<br /> int delimPosition = str.indexOf(delim, currentPosition);<br /> if (delimPosition >= 0 && delimPosition < nextDelimPosition) {<br /> nextDelimPosition = delimPosition;<br /> }<br /> String token = str.substring(currentPosition, nextDelimPosition);<br /> currentPosition = nextDelimPosition + 1;<br /> return token;<br /> }<br /><br /> public boolean hasMoreTokens() {<br /> return (currentPosition < str.length());<br /> }<br /> <br /> public int countTokens() {<br /> int count = 0;<br /> <br /> NonCollapsingStringTokenizer clone = (NonCollapsingStringTokenizer) this.clone();<br /> while(clone.hasMoreTokens())<br /> {<br /> clone.nextToken();<br /> count++;<br /> }<br /> <br /> return count;<br /> }<br /> <br /> public Object clone() {<br /> NonCollapsingStringTokenizer copy = new NonCollapsingStringTokenizer();<br /> copy.str = str;<br /> copy.delim = delim;<br /> return currentPosition;<br /> <br /> }<br />}<br />

The class above handles the challenge accordingly. This can be tested with the code below:

view plaincopy to clipboardprint?

1. public static void main(String[] args) {  
2.         String s = ",,dsafdf,,15,,,,";  
3.           
4.         final String _DELIM = ",";  
5.         //s = "token1,token2,token3,token4,token5,token6,token7,token8,token9";  
6.         s = ",,sffdsf,,bsgemail,1,1,5,dsf";  
7.                   
8.         NonCollapsingStringTokenizer ncst = new NonCollapsingStringTokenizer(s, _DELIM);  
9.         int x = 1;  
10.         while(ncst.hasMoreTokens()) {  
11.             String nextToken = ncst.nextToken();  
12.               
13.             System.out.println(Integer.toString(x++) + " " + nextToken);  
14.         }  
15.     }  

public static void main(String[] args) {<br /> String s = ",,dsafdf,,15,,,,";<br /> <br /> final String _DELIM = ",";<br /> //s = "token1,token2,token3,token4,token5,token6,token7,token8,token9";<br /> s = ",,sffdsf,,bsgemail,1,1,5,dsf";<br /> <br /> NonCollapsingStringTokenizer ncst = new NonCollapsingStringTokenizer(s, _DELIM);<br /> int x = 1;<br /> while(ncst.hasMoreTokens()) {<br /> String nextToken = ncst.nextToken();<br /> <br /> System.out.println(Integer.toString(x++) + " " + nextToken);<br /> }<br /> }

This produces the listing below:

view plaincopy to clipboardprint?

1. 1   
2. 2   
3. 3 sffdsf  
4. 4   
5. 5 bsgemail  
6. 6 1  
7. 7 1  
8. 8 5  
9. 9 dsf  

1 <br />2 <br />3 sffdsf<br />4 <br />5 bsgemail<br />6 1<br />7 1<br />8 5<br />9 dsf<br />