Month: September 2024

VMware NSX Manager application crash with Out of Memory due to large ApiTracker table on corfu db

Recently we were facing a problem on NSX control plane that suddenly crashed and became degraded. Initially we were thinking this could be solved by restarting the manager which was showing the problem. However, after the restart the problem moved to one of the other two controllers. We went on with other restarts of the entire control plane but it did not work. We noticed that the affected service is ‘Proton’ and found an OOM heap dump automatically created after the service crash:

ls /image/core
-rw-------  1 uproton uproton   37 Sep 13 21:41 proton_oom.hprof.gz

Latest after 15 minutes the problem appeared again so we suspected a memory leak and opened a SR with Broadcom. After rather short time it was discovered that ApiTracker table on corfu DB (corfu is a distributed db system built around a shared log abstraction among the NSX controllers) has had more than 10 million rows. Actually it is supposed to contain round about 1000 entries. We spent good amount of time to understand the reason since there should be an automatic cleanup job in place. This is how you can check the size of ApiTracker table:

tail -f /var/log/corfu/corfu-compactor-audit.log

Look for such output, especially for number of entries:

2024-09-13T23:23:45.020Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: completed checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b, entries(10030616), cpSize(2228864537) bytes at snapshot Token(epoch=1518, sequence=8128115261) in 3685860 ms

This entries are increasing because due to a bug a different table (EntityDeletionMarker) contains invalid entries. This can be if you upgraded the environment from older versions. GSS TSEs identified that this is preventing the execution of an automatic cleanup job of ApiTracker table.
This is how you can check if there are invalid entries on EntityDeletionMarker: 

./corfu_tool_runner.py --tool corfu-browser -n nsx -o showTable -t EntityDeletionMarker

After cleaning it up (do it only with GSS!) we were optimistic that the automatic job would kick in to clear the ApiTracker table. 

./corfu_tool_runner.py -n nsx -t EntityDeletionMarker -o clearTable

We were waiting two iterations (the job is starting every 15 minutes), however we noticed after 50 minutes it crashed and the node was again out of memory. So there was no other way than to increase the memory size of the controllers. This is unsupported and should be done only as last resort with GSS assistance. The memory can be increased by modifying this file:

/usr/tanuki/conf/proton-tomcat-wrapper_48gb.conf

And afterwards the controller need to be increased on virtual Machine level. In our case we put the Java Hep Size from 12% to 19.8%.

Finally, this fixed the issue and let the job complete the cleanup.

2024-09-13T22:22:19.068Z | INFO  |              Cmpt-chkpter-9000 |     o.c.runtime.view.SMRObject | ObjectBuilder: open Corfu stream nsx$ApiTracker id 84fde90f-0df5-305f-371b-ce8b5ca4267b
2024-09-13T22:22:19.158Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: Started checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b at snapshot Token(epoch=1518, sequence=8128115261)
2024-09-13T23:23:45.020Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: completed checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b, entries(10030616), cpSize(2228864537) bytes at snapshot Token(epoch=1518, sequence=8128115261) in 3685860 ms
2024-09-14T02:03:37.319Z | INFO  |              Cmpt-chkpter-9000 |     o.c.runtime.view.SMRObject | ObjectBuilder: open Corfu stream nsx$ApiTracker id 84fde90f-0df5-305f-371b-ce8b5ca4267b
2024-09-14T02:03:37.470Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: Started checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b at snapshot Token(epoch=1528, sequence=8130630447)
2024-09-14T02:18:35.227Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: completed checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b, entries(180000), cpSize(41796720) bytes at snapshot Token(epoch=1528, sequence=8130630447) in 897757 ms
2024-09-14T02:36:41.732Z | INFO  |              Cmpt-chkpter-9000 |     o.c.runtime.view.SMRObject | ObjectBuilder: open Corfu stream nsx$ApiTracker id 84fde90f-0df5-305f-371b-ce8b5ca4267b
2024-09-14T02:36:41.840Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: Started checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b at snapshot Token(epoch=1528, sequence=8130913177)
2024-09-14T02:36:43.122Z | INFO  |              Cmpt-chkpter-9000 |   o.c.runtime.CheckpointWriter | appendCheckpoint: completed checkpoint for 84fde90f-0df5-305f-371b-ce8b5ca4267b, entries(839), cpSize(209764) bytes at snapshot Token(epoch=1528, sequence=8130913177) in 1281 ms
2024-09-14T02:51:30.477Z | INFO  |              Cmpt-chkpter-9000 |     o.c.runtime.view.SMRObject | ObjectBuilder: open Corfu stream nsx$ApiTracker id 84fde90f-0df5-305f-371b-ce8b5ca4267b