git stderr: fatal: cannot exec '/tmp/appname/git-ssh.sh': Text file busy

[maidmantis]

Steps to reproduce

Can not possible to reproduce. It happens complete randomly (random time, random projects, random servers).
We use capistrano with Jenkins to deploy apps to servers.

Expected behavior

15:44:49 00:00 git:wrapper
15:44:49       01 mkdir -p /tmp/appname/
15:44:49     ✔ 01 deployer@10.10.10.10 0.259s
15:44:49       Uploading /tmp/appname/git-ssh.sh 100.0%
15:44:49       02 chmod +rx /tmp/appname/git-ssh.sh
15:44:49     ✔ 02 deployer@10.10.10.10 0.048s
15:44:49 00:00 git:check
15:44:49       01 git ls-remote --heads git@github.com:orgname/projectname.git

Actual behavior

13:33:31 00:00 git:wrapper
13:33:31       01 mkdir -p /tmp/appname/ 
13:33:31     ✔ 01 deployer@10.10.10.10 0.219s
13:33:31       Uploading /tmp/appname/git-ssh.sh 100.0%
13:33:31 00:00 git:check
13:33:31       01 git ls-remote --heads git@github.com:orgname/projectname.git
13:33:31       01 fatal: cannot exec '/tmp/appname/git-ssh.sh': Text file busy
13:33:31       01 fatal: unable to fork
13:33:31 (Backtrace restricted to imported tasks)
13:33:31 cap aborted!
13:33:31 SSHKit::Runner::ExecuteError: Exception while executing on host 10.10.10.10: git exit status: 128
13:33:31 git stdout: Nothing written
13:33:31 git stderr: fatal: cannot exec '/tmp/appname/git-ssh.sh': Text file busy
13:33:31 fatal: unable to fork
13:33:31 
13:33:31 
13:33:31 Caused by:
13:33:31 SSHKit::Command::Failed: git exit status: 128
13:33:31 git stdout: Nothing written
13:33:31 git stderr: fatal: cannot exec '/tmp/appname/git-ssh.sh': Text file busy
13:33:31 fatal: unable to fork
13:33:31 
13:33:31 Tasks: TOP => git:check
13:33:31 (See full trace by running task with --trace)

System configuration

cap <stage> doctor link

Additional Info

Found solution for similar issue:
just add bash command sync on remote server right after uploading instruction. It flushes all IO buffer to HDD.

[leehambley]

Any information you can provide about your OS environment/etc please?

[maidmantis]

Any information you can provide about your OS environment/etc please?

Hi @leehambley ,
Thanks for response.
All apps are running on AWS EC2 instances (types: t2.micro, t2.medium, m4.large, c5.large and EBSes gp2).
Following info is equal for all instances.

OS: Ubuntu 16.04.1 LTS
Kernel: Linux 10-10-10-10.internal 4.4.0-59-generic #80-Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
SSHD: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016

Best wishes

[will-in-wi]

Any chance you are doing multiple deploys to the same server? This sounds like a race condition where they are both contending for the same file.

Somewhat more esoteric possibilities might include the /tmp directory being mounted from a shared NFS server or something like that.

You could try adding set :tmp_dir, '/home/user/tmp' to use an application specific temp directory.

I don't think this will help, but according to the doctor output, you aren't running the latest version of Capistrano. Could you upgrade just to rule out any fixes we've already addressed?