"interrupted system call" error when calling putObject against samba-mounted minio

[ryangrahamnc]

I have a samba setup in kube. Kube node mounts samba as /samba (using automount). And minio pod mounts /samba/blah/minio to /data using hostPath.

And I'm getting interrupted system call errors on the server when writing files to minio using mc cp ~/Downloads/a.txt samba-async/test-bucket/a.txt.

The mc cmd succeeds in the end. But if I do the equivalent putObject command from nodejs's minio package, it throws S3Error: We encountered an internal error, please try again., which is difficult to program around.

This issue is very similar to #10402
But he was able to fix it up updating his kernel version. I updated to 5.4.0-52 and still have the problem.

I suspect the problem only happens when minio stat's at the end of putObject calls.

Expected Behavior

Write file and return success

Current Behavior

on client:
mc cp ~/Downloads/a.txt samba-async/test-bucket/a.txt
(succeeds on client)

server logs:

API: PutObject(bucket=test-bucket, object=a.txt)
Time: 14:35:45 UTC 11/08/2020
DeploymentID: d6341f50-7cb9-4637-86fd-034f85683529
RequestID: 16458F1113F0CD66
RemoteHost: 192.168.1.215
Host: 192.168.1.20:9003
UserAgent: MinIO (darwin; amd64) minio-go/v7.0.6 mc/2020-10-03T02:54:56Z
Error: stat /data/test-bucket/a.txt: interrupted system call (*os.PathError)
       5: cmd/fs-v1-helpers.go:210:cmd.fsStatFile()
       4: cmd/fs-v1.go:1242:cmd.(*FSObjects).putObject()
       3: cmd/fs-v1.go:1119:cmd.(*FSObjects).PutObject()
       2: cmd/object-handlers.go:1549:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2042:http.HandlerFunc.ServeHTTP()
API: PutObject(bucket=test-bucket, object=a.txt)
Time: 14:35:45 UTC 11/08/2020
DeploymentID: d6341f50-7cb9-4637-86fd-034f85683529
RequestID: 16458F1113F0CD66
RemoteHost: 192.168.1.215
Host: 192.168.1.20:9003
UserAgent: MinIO (darwin; amd64) minio-go/v7.0.6 mc/2020-10-03T02:54:56Z
Error: stat /data/test-bucket/a.txt: interrupted system call (*os.PathError)
       4: cmd/api-errors.go:1995:cmd.toAPIErrorCode()
       3: cmd/api-errors.go:2020:cmd.toAPIError()
       2: cmd/object-handlers.go:1551:cmd.objectAPIHandlers.PutObjectHandler()
       1: net/http/server.go:2042:http.HandlerFunc.ServeHTTP()

Also, when scrolling through the logs, I see more errors against stat calls.

Time: 14:12:05 UTC 11/08/2020
DeploymentID: d6341f50-7cb9-4637-86fd-034f85683529
Error: stat /data/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json: interrupted system call (*os.PathError)
       1: cmd/data-usage.go:59:cmd.storeDataUsageInBackend()
API: SYSTEM()
Time: 14:12:05 UTC 11/08/2020
DeploymentID: d6341f50-7cb9-4637-86fd-034f85683529
Error: stat /data/.minio.sys/buckets/.usage-cache.bin: interrupted system call (*os.PathError)
       6: cmd/fs-v1-helpers.go:210:cmd.fsStatFile()
       5: cmd/fs-v1.go:1242:cmd.(*FSObjects).putObject()
       4: cmd/fs-v1.go:1119:cmd.(*FSObjects).PutObject()
       3: cmd/data-usage-cache.go:466:cmd.(*dataUsageCache).save()
       2: cmd/fs-v1.go:306:cmd.(*FSObjects).CrawlAndGetDataUsage()
       1: cmd/data-crawler.go:108:cmd.runDataCrawler()
API: SYSTEM()
Time: 14:12:05 UTC 11/08/2020
DeploymentID: d6341f50-7cb9-4637-86fd-034f85683529
Error: stat /data/.minio.sys/buckets/.usage-cache.bin: interrupted system call (*os.PathError)
       2: cmd/fs-v1.go:306:cmd.(*FSObjects).CrawlAndGetDataUsage()
       1: cmd/data-crawler.go:108:cmd.runDataCrawler()
API: SYSTEM()
Time: 14:12:05 UTC 11/08/2020
DeploymentID: d6341f50-7cb9-4637-86fd-034f85683529
Error: stat /data/.minio.sys/buckets/.minio.sys/buckets/.usage.json/fs.json: interrupted system call (*os.PathError)
       1: cmd/data-usage.go:59:cmd.storeDataUsageInBackend()

If I exec into the container and do stat a.txt, it returns good info.

---

From nodejs:

var Minio = require('minio');
setTimeout(async()=>{
    try{
        var client = new Minio.Client({
            "endPoint": "192.168.1.20",
            "port": 9003,
            "useSSL": false,
            "accessKey": "adminUser",
            "secretKey": "pass"
        })
        var result = await client.fPutObject('test-bucket', 'a.txt', 'a.txt');
        console.log(1111, result)
    }catch(err){
        console.log('err', err);
    }
},1)

client error:

err S3Error: We encountered an internal error, please try again.
    at Object.parseError (/Users/ryan/crap/minio-samba-test/node_modules/minio/dist/main/xml-parsers.js:86:11)
    at /Users/ryan/crap/minio-samba-test/node_modules/minio/dist/main/transformers.js:156:22
    at DestroyableTransform._flush (/Users/ryan/crap/minio-samba-test/node_modules/minio/dist/main/transformers.js:80:10)
    at DestroyableTransform.prefinish (/Users/ryan/crap/minio-samba-test/node_modules/readable-stream/lib/_stream_transform.js:129:10)
    at DestroyableTransform.emit (events.js:315:20)
    at prefinish (/Users/ryan/crap/minio-samba-test/node_modules/readable-stream/lib/_stream_writable.js:611:14)
    at finishMaybe (/Users/ryan/crap/minio-samba-test/node_modules/readable-stream/lib/_stream_writable.js:620:5)
    at endWritable (/Users/ryan/crap/minio-samba-test/node_modules/readable-stream/lib/_stream_writable.js:643:3)
    at DestroyableTransform.Writable.end (/Users/ryan/crap/minio-samba-test/node_modules/readable-stream/lib/_stream_writable.js:571:22)
    at IncomingMessage.onend (_stream_readable.js:681:10) {
  code: 'InternalError',
  key: 'a.txt',
  bucketname: 'test-bucket',
  resource: '/test-bucket/a.txt',
  requestid: '1645929F12400A47',
  hostid: 'd6341f50-7cb9-4637-86fd-034f85683529',
  amzRequestid: null,
  amzId2: null,
  amzBucketRegion: null
}

If I exec into the container and do stat a.txt, it returns good info.
It also succeeds a good portion of the time as well.

Possible Solution

Dunno. It might not be a minio bug.
Could be samba, kernel, automount side effects, etc.

Context

My main issue is I cant do fputObject() calls from nodejs, since it throws an error some of the time.

Your Environment

minio version RELEASE.2020-11-06T23-17-07Z
mc version RELEASE.2020-10-03T02-54-56Z
ubuntu 18.04 bionic
kernel 5.4.0-52-generic
kube v1.15.1
docker 19.03.13
samba client 4.7.6
npm minio 7.0.16
I'm current using the minio admin user, so no access controls are causing this problem afaik

[harshavardhana]

This is a bug in CIFS samba which returns sigint for normal operations, nothing we can do here.

Alternatively use NFS

[ryangrahamnc]

Googled around some more, and it sounds like golang 1.16 will make this work with whatever magic they do.
Anyone who has this specific situation and wants to stay current can probably do a custom build with GODEBUG=asyncpreemptoff=1. I havent tried to see if it actually works though.
But for now, I've loaded in minio/minio:RELEASE.2020-06-18T02-23-35Z, which uses golang 1.13, and I'm not seeing any errors anymore.
For reference: restic/restic#2659