"Too Many Open Files" - Solve It Easy!
Application errors can sometimes be very frustrating to solve, more so in a situation where you cannot pinpoint its source. Below, you'll discover a few simple tips and tricks for resolving one error that isn't very frequent yet a show stopper when it does strike.
"Too many open files"
Sound familiar? If you are an application developer, you may break into a sweat at the very mention of this error. Most of applications, at some point in their life cycles – be it in development or testing or production – have come across this error. Some might argue about its frequency; but as an application developer, you would want to avoid this error at any cost.
~
Where does the error occur?
“Too many open files” can occur at any place where an external call is made from the application. The external call could be through http or tcp or native api calls to any external services such as
Caching services such as memcache and Redis.
NoSQL databases like Cassandra and MongoDB.
SQL databases including MySQL, Oracle, and MS SQL
File systems such as Dfs, Hadoop, or other file system calls
Potential causes of this error:
Some of the possible reasons for the “Too many open files” error could be
1. A system overload that could be caused~ by
- sudden surge in app usage caused by authorized users
- a malicious hacker attack on the application trying to bring it down
- failure of some app servers in a distributed deployment leading to overload in the remaining app servers
2. Closure of resourcesnot handled properly in the application code leading to open connections or files piling up.
3. Trying to connect to a non-existing host for an external service. For example, if the hostname url to connect to the external service is maintained in a database table and it has multiple entries for existing and non-existing hostnames, then the application will try to connect to all the entries in the database, thus ending up in connecting to a non-existing host, resulting in open files piling up.
Tips for resolving this error :
The usual tendency to overcome this error is to increase the open files limit at the system level. But that only defers the problem. Below are the various items that an application developer should check do to resolve the error.
- Configure the external services maximum open files or connections to the limit suggested by the service.
- Look for any redundant entries in the configuration file or database where the external service connection url is configured.
- If load is genuine, add more app servers.
- Use third party agents like http://file-leak-detector.kohsuke.org/ to detect open resources and to find the trace of the source that caused the file leak.
We had once faced such an issue in one of our test beds by having wrong entries connecting to the memcache server and had used this file leak detector tool to detect and resolve the error. We now are planning to add this feature to APM Insight. We shall keep this space updated once the feature is in place.
So the next time you face "Too many open files" error, you know what to do. Just follow these simple tips and don't lose your sleep over it.