Recently we converted our docker BuildGrid servers to run the storage backend OverlayFS (in docker the key config name is overlay). This provided us with great benefits - primarily around speed and reliability (barring this nasty bug that is!)
However we soon had an alert from our server monitoring framework that our root disk was running out of space - that was strange to us as we had mounted the /var/lib/docker/overlay folder onto a much larger separate drive. After some investigation, the culprit was identified as /var/lib/mlocate/mlocate.db - this file essentially powers the locate command; indexing disk paths; and providing quick file search results. Now that all sounds good.. our issue was that it was 9GB in size!
The underlying problem was caused by the fact that the overlay path was being indexed, and the nature of the union filesystem is such that there are a LOT of paths there (especially when you add lots of .git folders to the mix). As we didn't need this path to be included in the locate index, the solution we used was:
- Edit the
/etc/updatedb.conffile - Amend the line
PRUNEPATHS="...<<original content>>... /var/lib/docker". - Manually regenerate locate index file by running
sudo updatedb(eventually a cron task would have done this for us, but we wanted immediate results!)
This resulted in a new /var/lib/mlocate/mlocate.db file of 2MB in size... Win!