Clarify in installation instructions how much shared memory is needed (MacOS)

[ivalaginja]

The installation instructions in the readme of catkit2 just state that the default of 4 MB of shared memory on MacOS is not enough, but not how much is recommended (see https://github.com/spacetelescope/catkit2#shared-memory-configuration).

It seems like no action is needed for MacOS Catalina. I am on MacOS Monterey however, and from what I can tell its default is still at around 4 MB. I found this set of instructions how to deal with shared memory allocation on MacOS:
http://www.spy-hill.net/help/apple/SharedMemory.html
Edit: I also found this link useful: https://www.postgresql.org/docs/8.4/kernel-resources.html

And when I run sysctl -A | grep shm on my MacOS 12.2 (Monterey) it returns:

kern.sysv.shmall: 1024
kern.sysv.shmmax: 4194304
kern.sysv.shmmin: 1
kern.sysv.shmmni: 32
kern.sysv.shmseg: 8

The current value for kern.sysv.shmmax suggests a max of ~4MB if I see that right, is that correct?

The link above tells me how to change my shared memory settings but I don't know at this point what the recommended settings are.

@ehpor what would you suggest I try? If whatever I try works fine, I can update the readme accordingly.

[ehpor]

The largest memory buffer that we use on HiCAT is about 0.5 GB (2000x3000 px, 20 frames, float32). So try 1GB? I don't know if there are limits, or even why the (published) default on MacOS is so low compared to linux or windows. As I don't have a Mac, I cannot test this myself, so feel free to try out the HiCAT simulator on Mac and document any changes that you needed to make.

And yes to updating documentation. However, do it in the installation.rst file, so that it gets put on https://spacetelescope.github.io/catkit2/installation.html. I want to move the installation instructions from the readme into there.

[ivalaginja]

I am able to use the HiCAT simulator in its current form without having to do any changes to my local machine. I am not sure what that means, but I will keep an eye on this as I keep working with it.

Also, I found that the two links above don't apply to MacOS Monterey, and I would need more something like this:
https://unix.stackexchange.com/questions/689295/values-from-sysctl-a-dont-match-etc-sysctl-conf-even-after-restart

[raphaelpclt]

Some more information in here:
https://unix.stackexchange.com/questions/689295/values-from-sysctl-a-dont-match-etc-sysctl-conf-even-after-restart

[RemiSoummer]

@ivalaginja @ehpor , I'm reviving this discussion with @raphaelpclt.
I'm running Ventura 13.6 and been working on re-installing everything from scratch (as well as working on updating install documentation). Trying to run the GUI in simulation I have a AttributeError: Too many open files
which seems related to the discussion about sysctl above. I'm investigating how to increase that, and will put that in my PR documentation when we find the solution.

[ivalaginja]

@RemiSoummer you should pull @alexisyslau into the loop, she's been having exactly the same issue on her machine and has found a solution for it.

[RemiSoummer]

Thanks and yes I'm interested we are going in circles for now.

[RemiSoummer]

I just upgraded to Sonoma 14.4 and the problem is gone :-)
Still interested in getting the solution documented since that might likely come back.

For completeness this is what I have now:

$ sysctl -A | grep maxfiles

kern.maxfiles: 245760
kern.maxfilesperproc: 122880

and also:

$ sysctl -A | grep shm

kern.sysv.shmmax: 4194304
kern.sysv.shmmin: 1
kern.sysv.shmmni: 32
kern.sysv.shmseg: 8
kern.sysv.shmall: 1024
machdep.pmap.hashmax: 11
security.mac.posixshm_enforce: 1
security.mac.sysvshm_enforce: 1

[alexisyslau]

After testing with Iva's DataStream test, I think the problem is not with kern.maxfiles but with ulimit -n Perhaps some of you would have a better understanding of these settings.

From the tests I have conducted, ulimit -n limits how many files can be opened at once by the shell.

If you change ulimit -n to something higher (e.g., 1,000,000), you should be able to open 1,000,000 DataStreams without any issue. Currently, I do not know the upper limit for Mac.

A temporary (or semi-permanent) solution is to add: echo 'ulimit -n 1000000' to your shell profile (e.g., bash). Then ulimit -n would be changed each time you access the shell.

Sanity check to make sure the number of file descriptor has been changed would be: ulimit -n or ulimit -a (show other parameters and also their descriptions).

[ivalaginja]

For the record, the test, which should probably be committed to the repo:

from catkit2.catkit_bindings import DataStream
from tqdm import tqdm


streams = []

for i in tqdm(range(100000)):
    service_id = f'tessdft{i}'

    stream = DataStream.create('tester', service_id, 'int8', [1], 20)
    streams.append(stream)

    stream2 = DataStream.open(stream.stream_id)
    streams.append(stream2)

[ehpor]

@ivalaginja Agreed. Just a small detail: both the shared memory of the created and the opened DataStream count as an open file to the operating system. Depending on the operating system, the synchronization part of a DataStream can count as a few open files as well, but I'm not quite sure about the details there. (Semaphores may count as a file. They are at least open handles.)