Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test suite fails on macOS #133

Open
sternenseemann opened this issue Aug 6, 2021 · 8 comments
Open

Test suite fails on macOS #133

sternenseemann opened this issue Aug 6, 2021 · 8 comments
Labels
bug platform: macOS re: testsuite Conerning the test suite

Comments

@sternenseemann
Copy link 

Preprocessing test suite 'specs' for acid-state-0.16.0.1..
Building test suite 'specs' for acid-state-0.16.0.1..
[1 of 2] Compiling Data.Acid.TemplateHaskellSpec ( test/Data/Acid/TemplateHaskellSpec.hs, dist/build/specs/specs-tmp/Data/Acid/TemplateHaskellSpec.o, dist/build/specs/specs-tmp/Data/Acid/TemplateHaskellSpec.dyn_o )
[2 of 2] Compiling Main             ( test/Spec.hs, dist/build/specs/specs-tmp/Main.o, dist/build/specs/specs-tmp/Main.dyn_o )
Linking dist/build/specs/specs ...
Preprocessing test suite 'examples' for acid-state-0.16.0.1..
Building test suite 'examples' for acid-state-0.16.0.1..
[ 1 of 20] Compiling ChangeState      ( examples/errors/ChangeState.hs, dist/build/examples/examples-tmp/ChangeState.o, dist/build/examples/examples-tmp/ChangeState.dyn_o )
[ 2 of 20] Compiling ChangeVersion    ( examples/errors/ChangeVersion.hs, dist/build/examples/examples-tmp/ChangeVersion.o, dist/build/examples/examples-tmp/ChangeVersion.dyn_o )
[ 3 of 20] Compiling CheckpointCutsEvent ( examples/CheckpointCutsEvent.hs, dist/build/examples/examples-tmp/CheckpointCutsEvent.o, dist/build/examples/examples-tmp/CheckpointCutsEvent.dyn_o )
[ 4 of 20] Compiling Exceptions       ( examples/errors/Exceptions.hs, dist/build/examples/examples-tmp/Exceptions.o, dist/build/examples/examples-tmp/Exceptions.dyn_o )
[ 5 of 20] Compiling HelloDatabase    ( examples/HelloDatabase.hs, dist/build/examples/examples-tmp/HelloDatabase.o, dist/build/examples/examples-tmp/HelloDatabase.dyn_o )
[ 6 of 20] Compiling HelloWorld       ( examples/HelloWorld.hs, dist/build/examples/examples-tmp/HelloWorld.o, dist/build/examples/examples-tmp/HelloWorld.dyn_o )
[ 7 of 20] Compiling HelloWorldNoTH   ( examples/HelloWorldNoTH.hs, dist/build/examples/examples-tmp/HelloWorldNoTH.o, dist/build/examples/examples-tmp/HelloWorldNoTH.dyn_o )
[ 8 of 20] Compiling KeyValue         ( examples/KeyValue.hs, dist/build/examples/examples-tmp/KeyValue.o, dist/build/examples/examples-tmp/KeyValue.dyn_o )
[ 9 of 20] Compiling KeyValueNoTH     ( examples/KeyValueNoTH.hs, dist/build/examples/examples-tmp/KeyValueNoTH.o, dist/build/examples/examples-tmp/KeyValueNoTH.dyn_o )
[10 of 20] Compiling MonadStateConstraint ( examples/MonadStateConstraint.hs, dist/build/examples/examples-tmp/MonadStateConstraint.o, dist/build/examples/examples-tmp/MonadStateConstraint.dyn_o )
[11 of 20] Compiling ParameterisedState ( examples/ParameterisedState.hs, dist/build/examples/examples-tmp/ParameterisedState.o, dist/build/examples/examples-tmp/ParameterisedState.dyn_o )
[12 of 20] Compiling Proxy            ( examples/Proxy.hs, dist/build/examples/examples-tmp/Proxy.o, dist/build/examples/examples-tmp/Proxy.dyn_o )
[13 of 20] Compiling RemoteCommon     ( examples/RemoteCommon.hs, dist/build/examples/examples-tmp/RemoteCommon.o, dist/build/examples/examples-tmp/RemoteCommon.dyn_o )
[14 of 20] Compiling RemoteClient     ( examples/RemoteClient.hs, dist/build/examples/examples-tmp/RemoteClient.o, dist/build/examples/examples-tmp/RemoteClient.dyn_o )
[15 of 20] Compiling RemoteServer     ( examples/RemoteServer.hs, dist/build/examples/examples-tmp/RemoteServer.o, dist/build/examples/examples-tmp/RemoteServer.dyn_o )
[16 of 20] Compiling RemoveEvent      ( examples/errors/RemoveEvent.hs, dist/build/examples/examples-tmp/RemoveEvent.o, dist/build/examples/examples-tmp/RemoveEvent.dyn_o )
[17 of 20] Compiling SlowCheckpoint   ( examples/SlowCheckpoint.hs, dist/build/examples/examples-tmp/SlowCheckpoint.o, dist/build/examples/examples-tmp/SlowCheckpoint.dyn_o )
[18 of 20] Compiling Main             ( examples/Examples.hs, dist/build/examples/examples-tmp/Main.o, dist/build/examples/examples-tmp/Main.dyn_o )
[19 of 20] Compiling StressTest       ( examples/StressTest.hs, dist/build/examples/examples-tmp/StressTest.o, dist/build/examples/examples-tmp/StressTest.dyn_o )
[20 of 20] Compiling StressTestNoTH   ( examples/StressTestNoTH.hs, dist/build/examples/examples-tmp/StressTestNoTH.o, dist/build/examples/examples-tmp/StressTestNoTH.dyn_o )
Linking dist/build/examples/examples ...
Preprocessing executable 'acid-state-repair' for acid-state-0.16.0.1..
Building executable 'acid-state-repair' for acid-state-0.16.0.1..
[1 of 1] Compiling Main             ( repair/Main.hs, dist/build/acid-state-repair/acid-state-repair-tmp/Main.o )
Linking dist/build/acid-state-repair/acid-state-repair ...
Preprocessing test suite 'state-machine' for acid-state-0.16.0.1..
Building test suite 'state-machine' for acid-state-0.16.0.1..
[1 of 3] Compiling Data.Acid.StateMachineTest ( test/Data/Acid/StateMachineTest.hs, dist/build/state-machine/state-machine-tmp/Data/Acid/StateMachineTest.o, dist/build/state-machine/state-machine-tmp/Data/Acid/StateMachineTest.dyn_o )
[2 of 3] Compiling Data.Acid.KeyValueStateMachine ( test/Data/Acid/KeyValueStateMachine.hs, dist/build/state-machine/state-machine-tmp/Data/Acid/KeyValueStateMachine.o, dist/build/state-machine/state-machine-tmp/Data/Acid/KeyValueStateMachine.dyn_o )
[3 of 3] Compiling Main             ( test/StateMachine.hs, dist/build/state-machine/state-machine-tmp/Main.o, dist/build/state-machine/state-machine-tmp/Main.dyn_o )
Linking dist/build/state-machine/state-machine ...
running tests
Running 3 test suites...
Test suite specs: RUNNING...
Test suite specs: PASS
Test suite logged to: dist/test/acid-state-0.16.0.1-specs.log
Test suite examples: RUNNING...
Test suite examples: PASS
Test suite logged to: dist/test/acid-state-0.16.0.1-examples.log
Test suite state-machine: RUNNING...
━━━ Data.Acid.KeyValueStateMachine ━━━
  ○ 3/5 complete (running)
state-machine: getCurrentDirectory:getWorkingDirectory: resource exhausted (Too many open files)
Test suite state-machine: FAIL
Test suite logged to: dist/test/acid-state-0.16.0.1-state-machine.log
2 of 3 test suites (2 of 3 test cases) passed.
@ddssff
Copy link
Collaborator

ddssff commented Aug 7, 2021

I wonder what the limit is on macos?

@sternenseemann
Copy link
Author 

$ launchctl limit maxfiles
	maxfiles    256            unlimited

So seems the soft fd limit is quite low by default (256 on my system) which sounds plausible that you hit in the test suite.

@ddssff
Copy link
Collaborator

ddssff commented Aug 8, 2021

And yet...what changed? And why does this feel so familiar?

@sternenseemann
Copy link
Author

sternenseemann commented Aug 8, 2021 •
edited

I'm not sure what you are referring to. For what it's worth acid-state has been failing like this on darwin since May at least. Before that I have no data, sadly.

@ddssff
Copy link
Collaborator

ddssff commented Aug 8, 2021

The limit is 1024 on ubuntu, if I set ulimit -n 256 and run the test suite it fails.

@ddssff
Copy link
Collaborator

ddssff commented Aug 8, 2021 

    ━━━ Exception (IOException) ━━━
    state/KeyValueSequentialTest/open.lock: openFd: resource exhausted (Too many open files)
  
    This failure can be reproduced by running:
    > recheck (Size 42) (Seed 16729543741219666746 3391217788386191459) prop_sequential
  
  ✗ prop_parallel failed at test/Data/Acid/StateMachineTest.hs:493:15
    after 17 tests and 5 shrinks.
  
        ┏━━ test/Data/Acid/StateMachineTest.hs ━━━
    486 ┃ acidStateParallelProperty :: AcidStateInterface s -> Gen s -> Range Int -> Range Int -> [Command Gen (TestT IO) (Model s)] -> Property
    487 ┃ acidStateParallelProperty i gen_initial_state prefix_range parallel_range commands = property $ do
    488 ┃     actions <- forAll $ Gen.parallel prefix_range parallel_range StateAbsent $
    489 ┃                  [ open i gen_initial_state
    490 ┃                  , checkpoint i
    491 ┃                  ] ++ commands ++ commands
        ┃                  │ ━━━ Prefix ━━━
        ┃                  │ 
        ┃                  │ ━━━ Branch 1 ━━━
        ┃                  │ 
        ┃                  │ ━━━ Branch 2 ━━━
        ┃                  │ Var 0 = Open (KeyValue (fromList []))
    492 ┃     test $ do liftIO $ resetState i
    493 ┃               executeParallel StateAbsent actions
        ┃               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        ┃               │ ━━━ Exception (IOException) ━━━
        ┃               │ state/KeyValueParallelTest: getDirectoryContents:openDirStream: resource exhausted (Too many open files)
  
    This failure can be reproduced by running:
    > recheck (Size 16) (Seed 5671783275542411754 2683756560449981757) prop_parallel

@stepcut
Copy link
Member

stepcut commented Aug 9, 2021

I think the problem is that in this function,

https://github.com/acid-state/acid-state/blob/master/test/Data/Acid/StateMachineTest.hs#L486

it calls 'open' to open an acid-state test DB, but never calls close. So there is a file handle leak and that exceeds the limits on OS X. However, the way the 'parallel' function works in Hedgehog -- you can provide a prefix list of commands, but not a postfix list of commands. So it is not obvious how to close the handles.

Since this is just a test suite, the file handle leaks are perhaps ok, aside from the failure on OS X. Perhaps there is a way to limit the number of tests run to avoid hitting the limit on OS X?

@andreasabel andreasabel mentioned this issue Feb 14, 2022
Merged
@ddssff
Copy link
Collaborator

ddssff commented Feb 14, 2022

Increasing the limit on macos might be the easy thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug platform: macOS re: testsuite Conerning the test suite
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@stepcut @andreasabel @sternenseemann @ddssff