cPanel: Complete Backups Prevented by Default FTP Limits

Have you ever wondered if your cPanel backup is missing anything? We sure have and recently found out that we went many months without proper backups on another server of ours because the people at cPanel added default constraints to their software which prevents FTP programs from being able to see more than 10,000 files in any folder. This is by design from cPanel, they have been aware of this problem for years, and they continue to defend the practice. As a result, we had a folder with hundreds of thousands of images which we believed to have been backed up due to us regularly logging into FileZilla and directing it to download every file in the home directory. Turns out it only downloaded the first 10,000 files per folder. We have since fixed the problem and are writing this so that you don't suffer the same fate.

UPDATE: We have learned that this article is only a temporary fix. It seems cPanel will likely restore the default setting eventually. We are in the process of trying to figure out why they restore the default and how to prevent it because otherwise we will have to repeat this process whenever we want to backup a website.

Why use cPanel? Because despite their flaws they really are the only way to stand a chance running a Linux server if like us you've only used Windows in the past. Unfortunately, cPanel likes to impose a lot of default constraints on your system. They typically will defend the constraints as not being adverse to most users or in some cases benefitting most users. That is case here. The people at cPanel think it is better to limit the number of files that can be read in a folder to keep the server from being slowed down by having to read all files. They think that if they automatically allowed FTP programs to list every file that it would slow servers down too much and they would get complaints from clients of theirs that provide hosting services. A Google search of this problem typically results in a mixture of users angry about their inability to download all their files and hosting providers angry about having to limit recursion to speed up performance. In the event that you find out any of your folders have more than 10,000 files in them then you have to jump through hoops just to fix the problem. This problem can only be fixed using a command line interface (CLI) which is a huge hurdle for most people new to Linux who've never needed to use a command line interface before. The CLI would not be necessary if cPanel simply added a way to change the LimitRecursion setting to the FTP Server Configuration page in WHM, but they didn't so you have to use a CLI.

To make matters worse, cPanel does not display accurate numbers in File Manager when it comes to folder size. We have a folder containing about 325,000 files totaling just over 20 GB but cPanel's File Manager claims the folder is just 11.4 MB in size. If you ask cPanel why, they would probably tell you that it takes time for the system to figure out how big the folder is and that they simply programmed it to run faster by not figuring that out. Most users would be better off if they did not do that. Now you have to jump through hoops just to know how big a folder is. In fact, the best advise we can give you when it comes to finding out the real size of a folder in cPanel is to download it to a Windows machine, right click on the folder, and select properties which will show you the file size. Surely the solution to this problem on the cPanel end if one exists likely requires some other CLI action which we would have to research and implement. Way too much work for something which should just show the real size of the folder where it is listed by default.

What Should You Do?

Set LimitRecursion to a high number far in excess of what you suspect the largest file count in any folder on your system to be. You can do that by logging into WHM, going to the Terminal, and typing "cd /etc/" without quotes. Then type "vi pure-ftpd.conf" to open the FTP configuration file. Find the line which begins with "LimitRecursion" change to insert mode by typing "i", change 10000 to something really high (in our case we changed it to 500000), hit esc to leave insert mode, and type ":w" to save your progress. Then reboot the server to make the change take effect.

What Should cPanel Do?

cPanel should disable the recursion limit by default and add an option to the FTP Server Configuration page in WHM so that users can enable it if they want to. It could just be a simple binary checkbox to enable limiting combined with a text input to set the limit. Requiring hosting services to take extra steps to limit their users makes more sense than limiting all users by default just in case they're a hosting provider or someone so speed sensitive that they're willing to risk losing work just so the rest can load faster. Then nobody will get blindsided by incomplete backups limited only by settings no reasonable person would think to look for.

Conclusion

While most default constraints imposed by cPanel are harmless, this one is extremely dangerous. People should at least receive a notice that they won't be able to download more than 10,000 files in a folder unless they do something but they get no feedback until they go looking for files that should be backed up but were not. It is only a matter of time until someone loses a lot of files due to incomplete backups courtesy of default constraints from cPanel.