After running connected components, there is a large amount of temporary memory left in the spark folders - mainly in "blockmgr-*" folders, as well as the last checkpoint (although that's much smaller). The blockmgr folders are deleted when spark shuts down, but checkpoint folder isn't. The spark GC doesn't otherwise seem to clean up the blockmgr, so if I run connected components several times it keeps growing linearly.
After running connected components, there is a large amount of temporary memory left in the spark folders - mainly in "blockmgr-*" folders, as well as the last checkpoint (although that's much smaller). The blockmgr folders are deleted when spark shuts down, but checkpoint folder isn't. The spark GC doesn't otherwise seem to clean up the blockmgr, so if I run connected components several times it keeps growing linearly.