java.util.ConcurrentModificationException--What it means

[Symptom from logcat]
11-26 22:02:37.859   421   487 D ConnectivityService: ConnectivityChange for mobile: DISCONNECTED/DISCONNECTED
11-26 22:02:37.859   421   682 D NetdConnector: RMV <- {200 39372 Route added to default table}
11-26 22:02:37.859   421   484 D StateMachine: handleMessage: E msg.what=131085
11-26 22:02:37.859   421   484 D StateMachine: processMsg: DriverUnloadedState
11-26 22:02:37.859   421   484 D WifiStateMachine: DriverUnloadedState{ what=131085 when=-1ms }
11-26 22:02:37.859   421   487 D ConnectivityService: Attempting to switch to WIFI
11-26 22:02:37.869   421   484 D StateMachine: processMsg: DefaultState
11-26 22:02:37.869   421   484 D WifiStateMachine: DefaultState{ what=131085 when=-5ms }
11-26 22:02:37.869   421   484 D StateMachine: handleMessage: X
11-26 22:02:37.869   421   484 D StateMachine: handleMessage: E msg.what=131147
11-26 22:02:37.869   421   484 D StateMachine: processMsg: DriverUnloadedState
11-26 22:02:37.869   421   484 D WifiStateMachine: DriverUnloadedState{ what=131147 when=-8ms }
11-26 22:02:37.869   421   484 D StateMachine: processMsg: DefaultState
11-26 22:02:37.869   421   484 D WifiStateMachine: DefaultState{ what=131147 when=-11ms }
11-26 22:02:37.869   421   484 D StateMachine: handleMessage: X
11-26 22:02:37.869   421   487 D ConnectivityService: Attempting to switch to BLUETOOTH_TETHER
11-26 22:02:37.869   421   487 D ConnectivityService: handleConnectivityChange: changed linkProperty[0]: doReset=true resetMask=3
11-26 22:02:37.869   421   487 D ConnectivityService:   curLp=InterfaceName: rmnet0 LinkAddresses: [192.168.35.117/30,] Routes: [0.0.0.0/0 -> 192.168.35.118,] DnsAddresses: [192.168.212.10,]
11-26 22:02:37.869   421   487 D ConnectivityService:   newLp= null
11-26 22:02:37.879   421   682 D ConnectivityService: Adding 192.168.23.5/32 -> 172.11.208.57 for interface rmnet1

11-26 22:02:37.919   499   499 D StatusBar.NetworkController: onDataConnectionStateChanged: state=0 type=0
11-26 22:02:37.929   499   499 E StatusBar.NetworkController: updateDataNetType NETWORK_TYPE_UNKNOWN
11-26 22:02:38.069   421   487 W dalvikvm: threadid=44: thread exiting with uncaught exception (group=0x40c07498)

11-26 22:02:38.149   421   487 E AndroidRuntime: *** FATAL EXCEPTION IN SYSTEM PROCESS: ConnectivityServiceThread
11-26 22:02:38.149   421   487 E AndroidRuntime: java.util.ConcurrentModificationException
11-26 22:02:38.149   421   487 E AndroidRuntime:  at java.util.ArrayList$ArrayListIterator.next(ArrayList.java:569)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.modifyRoute(ConnectivityService.java:1553)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.modifyRoute(ConnectivityService.java:1528)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.removeRoute(ConnectivityService.java:1476)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.updateRoutes(ConnectivityService.java:2252)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.handleConnectivityChange(ConnectivityService.java:2179)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.handleDisconnect(ConnectivityService.java:1824)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService.access$1100(ConnectivityService.java:128)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at com.android.server.ConnectivityService$MyHandler.handleMessage(ConnectivityService.java:2687)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at android.os.Handler.dispatchMessage(Handler.java:99)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at android.os.Looper.loop(Looper.java:137)
11-26 22:02:38.149   421   487 E AndroidRuntime:  at android.os.HandlerThread.run(HandlerThread.java:60)
11-26 22:02:38.189   421   487 W DropBoxManagerService: Dropping: system_server_crash (1098 > 0 bytes)
11-26 22:02:38.209   421   487 E ActivityManager: handleApplicationCrash happeded!! and crash level = 2, trigger panic!!
11-26 22:02:38.209   421   487 E ActivityManager: TriggerKernelPanic IN

[Analysis]
modifyRoute(...) @ ConnectivityService.java
1505    private boolean modifyRoute(String ifaceName, LinkProperties lp, RouteInfo r, int cycleCount,
1506            boolean doAdd, boolean toDefaultTable) {
1507        if ((ifaceName == null) || (lp == null) || (r == null)) {
1508            if (DBG) log("modifyRoute got unexpected null: " + ifaceName + ", " + lp + ", " + r);
1509            return false;
1510        }
1511
1512        if (cycleCount > MAX_HOSTROUTE_CYCLE_COUNT) {
1513            loge("Error modifying route - too much recursion");
1514            return false;
1515        }
1516
1517        if (r.isHostRoute() == false) {
1518            RouteInfo bestRoute = RouteInfo.selectBestRoute(lp.getRoutes(), r.getGateway());
1519            if (bestRoute != null) {
1520                if (bestRoute.getGateway().equals(r.getGateway())) {
1521                    // if there is no better route, add the implied hostroute for our gateway
1522                    bestRoute = RouteInfo.makeHostRoute(r.getGateway());
1523                } else {
1524                    // if we will connect to our gateway through another route, add a direct
1525                    // route to it's gateway
1526                    bestRoute = RouteInfo.makeHostRoute(r.getGateway(), bestRoute.getGateway());
1527                }
1528                modifyRoute(ifaceName, lp, bestRoute, cycleCount+1, doAdd, toDefaultTable);
1529            }
1530        }
1531        if (doAdd) {
1532            if (VDBG) log("Adding " + r + " for interface " + ifaceName);  ----- logout line "11-26 22:02:37.879   421   682 D ConnectivityService: Adding 192.168.23.5/32 -> 172.11.208.57 for interface rmnet1"
1533            try {
1534                if (toDefaultTable) {
1535                    mAddedRoutes.add(r);  // only track default table - only one apps can effect
1536                    if (VDBG) log("Routes in main table - [ " + mAddedRoutes + " ]");
1537                    mNetd.addRoute(ifaceName, r);
1538                } else {
1539                    mNetd.addSecondaryRoute(ifaceName, r);
1540                }
1541            } catch (Exception e) {
1542                // never crash - catch them all
1543                if (DBG) loge("Exception trying to add a route: " + e);
1544                return false;
1545            }
1546        } else {
1547            // if we remove this one and there are no more like it, then refcount==0 and
1548            // we can remove it from the table
1549            if (toDefaultTable) {
1550                // modified begin
1551                //mAddedRoutes.remove(r);
1552                for (Iterator<RouteInfo> it = mAddedRoutes.iterator(); it.hasNext();) {
1553                    RouteInfo thisRI = it.next();
1554                    if ( thisRI.equals(r) ) {
1555                        it.remove();
1556                    }
1557                }
1558                // modified end
1559                if (VDBG) log("Routes in main table - [ " + mAddedRoutes + " ]");
1560                if (mAddedRoutes.contains(r) == false) {
1561                    if (VDBG) log("Removing " + r + " for interface " + ifaceName);
1562                    try {
1563                        mNetd.removeRoute(ifaceName, r);
1564                    } catch (Exception e) {
1565                        // never crash - catch them all
1566                        if (VDBG) loge("Exception trying to remove a route: " + e);
1567                        return false;
1568                    }
1569                } else {
1570                    if (VDBG) log("not removing " + r + " as it's still in use");
1571                }
1572            } else {
1573                if (VDBG) log("Removing " + r + " for interface " + ifaceName);
1574                try {
1575                    mNetd.removeSecondaryRoute(ifaceName, r);
1576                } catch (Exception e) {
1577                    // never crash - catch them all
1578                    if (VDBG) loge("Exception trying to remove a route: " + e);
1579                    return false;
1580                }
1581            }
1582        }
1583        return true;
1584    }

We can see that adding a new item into mAddedRoutes in thread 682 and remove a item from mAddedRoutes in thread 487 at same time.

[JRE document Reference]
public class ConcurrentModificationException
extends RuntimeException
This exception may be thrown by methods that have detected concurrent modification of an object when such modification is not permissible.

For example, it is not generally permssible for one thread to modify a Collection while another thread is iterating over it. In general, the results of the iteration are undefined under these circumstances. Some Iterator implementations (including those of all the collection implementations provided by the JRE) may choose to throw this exception if this behavior is detected. Iterators that do this are known as fail-fast iterators, as they fail quickly and cleanly, rather that risking arbitrary, non-deterministic behavior at an undetermined time in the future.

Note that this exception does not always indicate that an object has been concurrently modified by a different thread. If a single thread issues a sequence of method invocations that violates the contract of an object, the object may throw this exception. For example, if a thread modifies a collection directly while it is iterating over the collection with a fail-fast iterator, the iterator will thow this exception.
Note that fail-fast behavior cannot be guaranteed as it is, generally speaking, impossible to make any hard guarantees in the presence of unsynchronized concurrent modification. Fail-fast operations throw ConcurrentModificationException on a best-effort basis. Therefore, it would be wrong to write a program that depended on this exception for its correctness: ConcurrentModificationException should be used only to detect bugs.

The modification status of a container will be recored during one iterator of this container being initialized.
Any modification that is not operated by that iterator of this container will lead to a "java.util.ConcurrentModificationException" if the iterator is still valid and active.

[Reference solution]
Adding a synchronization mechanism for mAddedRoutes such as synchronized(mAddedRoutes) to keep its modification thread-safe.