logcabin client crash when out of fds at fork

[nhardt]

logcabin has a philosophy that since you have to be able to recovery quickly from a failure, you should fail early and often. in accordance with this philosophy, if the logcabin process is out of file descriptors and an open() fails, it will abort. the shared code that the client uses has the same behavior. i don't know if the logcabin failure philosophy should also apply to clients.

the particular crash is here:

#0 0x00007f90ce0185f7 in raise () from /lib64/libc.so.6 
#1 0x00007f90ce019ce8 in abort () from /lib64/libc.so.6 
#2 0x0000000000aac6bb in LogCabin::Core::Random::(anonymous namespace)::RandomState::reset (this=0x138fea0 <LogCabin::Core::Random::(anonymous namespace)::randomState>) at build/Core/Random.cc:81 
#3 0x0000000000aac88a in LogCabin::Core::Random::(anonymous namespace)::resetRandomState () at build/Core/Random.cc:159 
#4 0x00007f90ce0a10e7 in fork () from /lib64/libc.so.6

    void reset() { 
        std::lock_guard<std::mutex> lockGuard(mutex); 
        int fd = open("/dev/urandom", O_RDONLY); 
        if (fd < 0) { 
            // too early to call PANIC in here 
            fprintf(stderr, "Couldn't open /dev/urandom: %s\n", 
                    strerror(errno)); 
            abort();

in this case, a client side fork() is also required.

possible fixes:

• up the fd limit for the client
• fix the log cabin client code to fall back to a random number generator that does not require a fd
• ensure the client isn't doing anything that is needlessly eating up file descriptors

i'm ok to close this as "won't fix" if the behavior is in line with that is expected or to implement a fix for this particular abort-on-fork when out of FDs issues by falling back to a different RNG.