With an un-explained exception for parsing 'bunny.gts', we do two further tests. First, we just tried to concatenate a string with itself repeatedly. At some given point, the jvm stalls. We also tried to read some key health indicators of the jvm. Please note the readings may differ as per the initiall JVM settings.

package net.heteroclinic.jgts.donotfix; public class TestMaxStringSizeDefaultEclipseSetting { public static void main(String[] args) { //http://viralpatel.net/blogs/getting-jvm-heap-size-used-memory-total-memory-using-java-runtime/ Runtime runtime = Runtime.getRuntime(); String str = "\na a a\n b b b"; int totalSize = 0; int i = 0; totalSize = str.length(); try { for ( ; i < Integer.MAX_VALUE; i++) { System.out.println("i = "+i); System.out.println("runtime.totalMemory() : " + runtime.totalMemory() ); System.out.println("runtime.freeMemory() : " + runtime.freeMemory() ); System.out.println("Sum of all strings' size = " + totalSize); str = str + str; System.out.println("New size of str = "+str.length() + "\n"); totalSize += str.length(); } } catch (Exception e) { e.printStackTrace(); } } } /* Default Setting in Eclipse renders: i = 0 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 13 New size of str = 26 i = 1 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 39 New size of str = 52 i = 2 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 91 New size of str = 104 i = 3 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 195 New size of str = 208 i = 4 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 403 New size of str = 416 i = 5 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 819 New size of str = 832 i = 6 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 1651 New size of str = 1664 i = 7 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 3315 New size of str = 3328 i = 8 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 6643 New size of str = 6656 i = 9 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 13299 New size of str = 13312 i = 10 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 26611 New size of str = 26624 i = 11 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121350688 Sum of all strings' size = 53235 New size of str = 53248 i = 12 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121084296 Sum of all strings' size = 106483 New size of str = 106496 i = 13 runtime.totalMemory() : 122683392 runtime.freeMemory() : 119912016 Sum of all strings' size = 212979 New size of str = 212992 i = 14 runtime.totalMemory() : 122683392 runtime.freeMemory() : 118846904 Sum of all strings' size = 425971 New size of str = 425984 i = 15 runtime.totalMemory() : 122683392 runtime.freeMemory() : 116716832 Sum of all strings' size = 851955 New size of str = 851968 i = 16 runtime.totalMemory() : 122683392 runtime.freeMemory() : 112456840 Sum of all strings' size = 1703923 New size of str = 1703936 i = 17 runtime.totalMemory() : 122683392 runtime.freeMemory() : 103937008 Sum of all strings' size = 3407859 New size of str = 3407872 i = 18 runtime.totalMemory() : 122683392 runtime.freeMemory() : 104668576 Sum of all strings' size = 6815731 New size of str = 6815744 i = 19 runtime.totalMemory() : 154664960 runtime.freeMemory() : 112197192 Sum of all strings' size = 13631475 New size of str = 13631488 i = 20 runtime.totalMemory() : 199229440 runtime.freeMemory() : 129587016 Sum of all strings' size = 27262963 New size of str = 27262976 i = 21 runtime.totalMemory() : 254279680 runtime.freeMemory() : 48322224 Sum of all strings' size = 54525939 New size of str = 54525952 i = 22 runtime.totalMemory() : 528482304 runtime.freeMemory() : 49894936 Sum of all strings' size = 109051891 New size of str = 109051904 i = 23 runtime.totalMemory() : 1075314688 runtime.freeMemory() : 51467648 Sum of all strings' size = 218103795 New size of str = 218103808 i = 24 runtime.totalMemory() : 1483210752 runtime.freeMemory() : 171932696 Sum of all strings' size = 436207603 Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421) at java.lang.StringBuilder.append(StringBuilder.java:136) at net.heteroclinic.jgts.TestMaxStringSize.main(TestMaxStringSize.java:27) */

This is a situation not easy to explain from the root and probably even more difficult to give a solution. But, what we observed is that if we concatenate large strings with the '+' operator, the process-ability of the StringBuilder module stressed out. This is the first situation we should try to avoid. It also shows the strings manipulation functions produce quite some by-products so the resources drained to an unkown sink.

Next, we try to gauge the size of the string we can split. Though such measure is not well defined, but we still see that the cost grows exponentially, at least it is non-linearly increasing, as the string size doubles. So far, using Regex to do file parsing seems to be off track.

package net.heteroclinic.jgts.donotfix; import java.util.Calendar; public class TestMaxStringSizeDefaultEclipseSettingSplit { public static void main(String[] args) { //http://viralpatel.net/blogs/getting-jvm-heap-size-used-memory-total-memory-using-java-runtime/ Runtime runtime = Runtime.getRuntime(); String str = "\na a a\n b b b"; int totalSize = 0; int i = 0; totalSize = str.length(); try { for ( ; i < Integer.MAX_VALUE; i++) { System.out.println("i = "+i); System.out.println("runtime.totalMemory() : " + runtime.totalMemory() ); System.out.println("runtime.freeMemory() : " + runtime.freeMemory() ); System.out.println("Sum of all strings' size = " + totalSize); str = str + str; System.out.println("New size of str = "+str.length()); long startTime = Calendar.getInstance().getTimeInMillis(); totalSize += str.length(); String [] arr = str.split("\\s+"); System.out.println(("Split to "+ arr.length) + " sub strings. "+ "It takes "+(Calendar.getInstance().getTimeInMillis() - startTime)); System.out.println(); } } catch (Exception e) { e.printStackTrace(); } } } /* When i = 22, it took minutes and still running. So we force stopping. i = 0 runtime.totalMemory() : 122683392 runtime.freeMemory() : 121403952 Sum of all strings' size = 13 New size of str = 26 Split to 13 sub strings. It takes 10 i = 1 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 39 New size of str = 52 Split to 25 sub strings. It takes 0 i = 2 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 91 New size of str = 104 Split to 49 sub strings. It takes 0 i = 3 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 195 New size of str = 208 Split to 97 sub strings. It takes 1 i = 4 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 403 New size of str = 416 Split to 193 sub strings. It takes 0 i = 5 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 819 New size of str = 832 Split to 385 sub strings. It takes 1 i = 6 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 1651 New size of str = 1664 Split to 769 sub strings. It takes 1 i = 7 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 3315 New size of str = 3328 Split to 1537 sub strings. It takes 2 i = 8 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 6643 New size of str = 6656 Split to 3073 sub strings. It takes 2 i = 9 runtime.totalMemory() : 122683392 runtime.freeMemory() : 120108232 Sum of all strings' size = 13299 New size of str = 13312 Split to 6145 sub strings. It takes 3 i = 10 runtime.totalMemory() : 122683392 runtime.freeMemory() : 119441968 Sum of all strings' size = 26611 New size of str = 26624 Split to 12289 sub strings. It takes 5 i = 11 runtime.totalMemory() : 122683392 runtime.freeMemory() : 118113520 Sum of all strings' size = 53235 New size of str = 53248 Split to 24577 sub strings. It takes 10 i = 12 runtime.totalMemory() : 122683392 runtime.freeMemory() : 116735896 Sum of all strings' size = 106483 New size of str = 106496 Split to 49153 sub strings. It takes 19 i = 13 runtime.totalMemory() : 122683392 runtime.freeMemory() : 112681904 Sum of all strings' size = 212979 New size of str = 212992 Split to 98305 sub strings. It takes 13 i = 14 runtime.totalMemory() : 122683392 runtime.freeMemory() : 105039864 Sum of all strings' size = 425971 New size of str = 425984 Split to 196609 sub strings. It takes 55 i = 15 runtime.totalMemory() : 122683392 runtime.freeMemory() : 109474728 Sum of all strings' size = 851955 New size of str = 851968 Split to 393217 sub strings. It takes 55 i = 16 runtime.totalMemory() : 154664960 runtime.freeMemory() : 121942712 Sum of all strings' size = 1703923 New size of str = 1703936 Split to 786433 sub strings. It takes 50 i = 17 runtime.totalMemory() : 154664960 runtime.freeMemory() : 63716224 Sum of all strings' size = 3407859 New size of str = 3407872 Split to 1572865 sub strings. It takes 433 i = 18 runtime.totalMemory() : 279969792 runtime.freeMemory() : 173400792 Sum of all strings' size = 6815731 New size of str = 6815744 Split to 3145729 sub strings. It takes 615 i = 19 runtime.totalMemory() : 446693376 runtime.freeMemory() : 237715280 Sum of all strings' size = 13631475 New size of str = 13631488 Split to 6291457 sub strings. It takes 1848 i = 20 runtime.totalMemory() : 747110400 runtime.freeMemory() : 316800696 Sum of all strings' size = 27262963 New size of str = 27262976 Split to 12582913 sub strings. It takes 4694 i = 21 runtime.totalMemory() : 1190133760 runtime.freeMemory() : 385191104 Sum of all strings' size = 54525939 New size of str = 54525952 Split to 25165825 sub strings. It takes 21443 i = 22 runtime.totalMemory() : 1806696448 runtime.freeMemory() : 259442936 Sum of all strings' size = 109051891 New size of str = 109051904 .... */

In fact, if we are using BufferedReader, the program will process data "in memory" behind the scene.

static GTSModel readWithLineReaderAGTSFile(String fileName) { return readAGTSFile(new File(fileName)); } static GTSModel readWithLineReaderAGTSFile(File file) { boolean succeeded = false; GTSModel model = new GTSModel(); URL fileURL = GTSFileReaderTest.class.getClassLoader().getResource(file.getName()); /* int totalVertices = 0, totalEdges= 0, totalTriangles= 0; float [][] vertices = null; int[][] edges = null; int[][] triangles = null; */ FileInputStream fis = null; BufferedReader br = null; try { fis = new FileInputStream(new File(fileURL.toURI())); //Construct BufferedReader from InputStreamReader br = new BufferedReader(new InputStreamReader(fis)); //Read the first line, to get totalVertices, totalEdges, totalTriangles String line = br.readLine(); if (null != line ) { String strFirstLine = line; String[] indices = strFirstLine.split("\\s"); try { model.totalVertices = Integer.parseInt(indices[0]); model.totalEdges = Integer.parseInt(indices[1]); model.totalTriangles = Integer.parseInt(indices[2]); } catch (Exception e) { System.out.println("Fail to parse header."); //throw again to goto disaster handle. throw new Exception("Fail to parse header."); } } else { System.out.println("Fail to read the first line."); throw new Exception("Fail to read the first line."); } // vertex (x,y,z); edge (beginVertexIndex, endVertexIndex); triangle (edgeOneIndex,edgeTwoIndex,edgeThreeIndex) //End read the first line model.vertices = new float[model.totalVertices][3] ; model.edges = new int[model.totalEdges][2] ; model.triangles = new int[model.totalTriangles][3] ; { //Read the vertices (geometry) int count = 0; for (int i = 0 ; i < model.totalVertices; i++) { try { line = br.readLine(); String [] arr = line.split("\\s"); model.vertices[i][0] = Float.parseFloat(arr[0]); count++; model.vertices[i][1] = Float.parseFloat(arr[1]); count++; model.vertices[i][2] = Float.parseFloat(arr[2]); count++; } catch (Exception e) { System.out.println("Fail to parse vertex at row "+(i)); //throw again to goto disaster handle. throw new Exception("Fail to parse vertex at row "+(i)); } } if (count != model.totalVertices * 3) { throw new Exception("Read vertices "); } model.atomicVerticesReadCount = count; //Read the vertices (geometry) } { //TO-DO metaphorically read the edges the same way int count = 0; for (int i = 0 ; i < model.totalEdges; i++) { try { line = br.readLine(); String [] arr = line.split("\\s"); model.edges[i][0] = Integer.parseInt(arr[0]); count++; model.edges[i][1] = Integer.parseInt(arr[1]); count++; } catch (Exception e) { System.out.println("Fail to parse edge at edge row "+(i)); //throw again to goto disaster handle. throw new Exception("Fail to parse edge at edge row "+(i)); } } if (count != model.totalEdges * 2) { throw new Exception("Read edges "); } model.atomicEdgesReadCount = count; } { //TO-DO metaphorically read the triangles int count = 0; for (int i = 0 ; i < model.totalTriangles; i++) { try { line = br.readLine(); String [] arr = line.split("\\s"); model.triangles[i][0] = Integer.parseInt(arr[0]); count++; model.triangles[i][1] = Integer.parseInt(arr[1]); count++; model.triangles[i][2] = Integer.parseInt(arr[2]); count++; } catch (Exception e) { System.out.println("Fail to parse triangle at triangle row "+(i)); //throw again to goto disaster handle. throw new Exception("Fail to parse triangle at triangle row "+(i)); } } if (count != model.totalTriangles * 3) { throw new Exception("Read triangles "); } model.atomicTrianglesReadCount = count; } succeeded =true; } catch (FileNotFoundException e) { e.printStackTrace(); } catch (IOException e) { e.printStackTrace(); } catch (URISyntaxException e) { e.printStackTrace(); } catch (Exception eat) { eat.printStackTrace(); } finally { if (null != br) { try { br.close(); } catch (IOException e) { e.printStackTrace(); } } if (null != fis) { try { fis.close(); } catch (IOException e) { e.printStackTrace(); } } } if (! (succeeded ) ) { return null; } else { return model; } }

We again add one more unit test to make sure future changes will not bring stall to the existing suite.

@Test public void testReadWithLineReaderAGTSFileBunnyForRegression() { boolean succeeded = false; final String readFileName = "bunny.gts"; URL fileURL = GTSFileReaderTest.class.getClassLoader().getResource(readFileName); File readFile; GTSModel bunny = null; try { readFile = new File(fileURL.toURI()); bunny = GTSFileReader.readWithLineReaderAGTSFile(readFile); //bunny.print(); succeeded = true; } catch (URISyntaxException e) { e.printStackTrace(); }catch (Exception e) { e.printStackTrace(); } //34834 104288 69451 if ( (! succeeded ) || (null == bunny)) { fail("Read "+readFileName+" failed."); } else { assertTrue("Fail to validate read",bunny.validateRead()); assertTrue("totalVertices wrong",bunny.totalVertices == 34834); assertTrue("totalTriangles wrong",bunny.totalTriangles == 69451); assertTrue("totalEdges wrong",bunny.totalEdges == 104288); } }