Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory not cleaned up #38

Open
BWorld opened this issue Dec 29, 2015 · 0 comments
Open

Memory not cleaned up #38

BWorld opened this issue Dec 29, 2015 · 0 comments

Comments

@BWorld
Copy link

BWorld commented Dec 29, 2015

Hi,

I think I have discovered an issue that the shared memory segments are not cleaned after a script is killed using Ctrl+C (SIGNINT?).
I have tried to isolate the issue as much as possible.

Background information about the real application, you can skip this part ^^
I approximately spawn 60 forks. Each forked process will register a gearman function to handle. (Don't worry in the test I left out gearman). Whenever the gearman deamon pushes a task to one of the 60 listeners it spawns another short living process which will handle the task (i.e. resize some image) and the process is terminated automatically which is a huge difference compared to the listeners which will never exit.

After running some unit tests for my application which trigger many background jobs an exception is thrown: when shmop_open() fails: 'Not able to create shared memory segment for PID: %s'

The isolated test behavior explained:
I have written a small isolated test that simulates the real applications behavior:

  • Open N number of job listeners per job type, each represents a separate process
  • When a listener receives a job it spawns a new process within the sub process of the listener
  • The job is executed, i.e. resizing an image (now faked with a usleep(1000))
  • The worker process is closed by reaching the end of the function
  • The listener is not blocked by the sub process'es wait() method anymore and returns to the start of the infinite loop

How to reproduce:

1. ipcs -m && ipcs -m | wc
Notice the number at the bottom. In my case it is 207.

ipcs -m && ipcs -m | wc

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 294912     cblokland  600        524288     2          dest         
.....
0x00000000 44892161   cblokland  600        393216     2          dest         


    207    1249   16360

2. phpunit --bootstrap bootstrap.php Spork/Test/NestedProcessTest.php
Let it run a while and continue to step 3 before quitting with Ctrl+C

3. ipcs -m && ipcs -m | wc
Notice the number has increased with approximately 6 or 8 (which is ok I assume? the code only spawns 4 processes?)

ipcs -m && ipcs -m | wc

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 294912     cblokland  600        524288     2          dest         
.....                      
0x00006224 490045650  cblokland  644        71         0                       

    215    1297   17000

4. stop the test script with Ctrl+C which causes a SIGINT to be sent

5. run again: ipcs -m && ipcs -m | wc
I expected to see 207 again but it stays at 212..

ipcs -m && ipcs -m | wc

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x00000000 294912     cblokland  600        524288     2          dest         
.....
0x00006bce 646086866  cblokland  644        71         0                       

    212    1279   16760

Why does it stay at 212 and doesn't return to 207 because that is what I expected.
Another curious thing is that when counting the number of total 'listener' processes I count 4 but 207 + 4 = 211 but somehow we end up having 212 memory segments?

Add this script to tests/Spork/Test/NestedProcessTest.php

<?php
namespace Spork\Test;

use Spork\ProcessManager;
use Spork\SharedMemory;

class NestedProcessTest extends \PHPUnit_Framework_TestCase
{

    protected $processManager;

    public function setUp()
    {
        $this->processManager = new ProcessManager();
        $this->processManager->setDebug(true);
    }

    public function tearDown()
    {
        unset($this->processManager);
    }

    public function testNestedProcesses()
    {
        $tasks = [
            // Imagine having image processors (resizing i.e.), we spawn N listeners
            'image_resizer' => 1,

            // Imagine having cache invalidators, we spawn N listeners again
            'cache_invalidator' => 3

            // etc etc
        ];

        foreach ($tasks as $taskId => $numChildren) {
            for ($i = 0; $i < $numChildren; $i ++) {
                $this->processManager->fork(new Listener());
            }
        }

        // Blocking because the listeners should never exit
        $this->processManager->wait();
    }
}

class Listener
{
    public function __invoke(SharedMemory $shm)
    {
        // Should never die and must have very light footprint. It will be just waiting for jobs
        // in real life code it will register a method at the gearman deamon for example.

        while (true) {
            // Fake we have received a job
            // $this->work() is blocking because of the ->wait() called and the worker process itself
            // has a usleep()
            $this->onJobReceived();
        }
    }

    protected function onJobReceived()
    {
        // Jobs are processed in their own fork and this fork must be killed as soon as the job has completed
        // i.e: the image is resized
        $processManager = new ProcessManager();
        $processManager->setDebug(true);

        // We will fake a worker here.
        $processManager->fork(new Worker());

        // Wait until the job has finished. (read: image is resized)
        $processManager->wait();
    }
}

class Worker
{
    public function __invoke(SharedMemory $shm)
    {
        // Fake that we are doing some work
        usleep(1000);
        return 0;
    }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant