Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot execute a sequential gsutil in windows batch file #233

Closed
zelon opened this issue Nov 1, 2014 · 12 comments
Closed

Cannot execute a sequential gsutil in windows batch file #233

zelon opened this issue Nov 1, 2014 · 12 comments

Comments

@zelon
Copy link

zelon commented Nov 1, 2014

I'm using gsutil to backup my datas in windows. And my batch file is like below:

gsutil cp -R a gs://zelon-test/
gsutil cp -R b gs://zelon-test/

But only the first command "gsutil cp -R a gs://zelon-test/" is executed. Second command is not executed at all.

@mfschwartz
Copy link
Contributor

Hi - did the first command exit with a non-0 exit status? That would cause the batch file to abort.

@zelon
Copy link
Author

zelon commented Nov 2, 2014

I've tested like below:

a.exe
echo Exit Code is %errorlevel%
gsutil cp -R a gs://zelon-test/
echo Exit Code is %errorlevel%
gsutil cp -R b gs://zelon-test/
echo Exit Code is %errorlevel%

below is the result:

C:\Temp>test.bat

C:\Temp>a.exe

C:\Temp>echo Exit Code is 0
Exit Code is 0

C:\Temp>gsutil cp -R a gs://zelon-test/
Copying file://a\find.txt [Content-Type=text/plain]...
Uploading gs://zelon-test/a/find.txt: 36 KB/36 KB

C:\Temp>

First echo command is executed and show the return value of a.exe. But second echo command(after first gsutil command) is not executed. It seems that gsutil kill the windows batch process.

@mfschwartz
Copy link
Contributor

I just tried running the following script on Windows 7 Ultimate with Service Pack 1, and it worked for me:

\Python27\python.exe \cygwin\usr\local\bin\gsutil\gsutil cp c:\try.bat gs://my-bucket/1
\Python27\python.exe \cygwin\usr\local\bin\gsutil\gsutil cp c:\try.bat gs://my-bucket/2


What version of Windows are you runing?
What happens if you change the commands to run gsutil using the full path to the python interpreter and to the gsutil script, similar to what I did above?

Mike

@zelon
Copy link
Author

zelon commented Nov 8, 2014

I've address this issue with your advice. Thank you.

I'm using Windows 8.1 Pro K(Korean) and default python version is 2.7.6 64 bit
And I tried to use python with full path as you mentioned. Then, some error message appeared like below:

Uploading gs://zelon-test/a.exe: 14 KB/14 KB
ERROR 1108 16:48:44.817000 http_wrapper.py] Response returned status 401, retrying
ERROR 1108 16:48:44.817000 http_wrapper.py] Retrying request to url https://www.googleapis.com/upload/XXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXuploadType=multipart after exception HttpError accessing <https://www.googleapis.com/upload/XXXXXXXXXXXXXXXXXXXXXXXXXX
fields=generation%2Ccrc32c%2Cmd5Hash%2Csize&alt=json&key=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX&prettyPrint=True&upload
Type=multipart>: response: <{'status': '401', 'alternate-protocol': '443:quic,p=0.01', 'content-length': '238', 'vary':
'Origin, Referer, X-Origin', 'server': 'UploadServer ("Built on Oct 29 2014 16:52:36 (1414626756)")', 'date': 'Sat, 08 N
ov 2014 07:48:45 GMT', 'content-type': 'application/json; charset=UTF-8', 'www-authenticate': 'Bearer realm="https://acc
ounts.google.com/AuthSubRequest"'}>, content <{
"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Login Required",
"locationType": "header",
"location": "Authorization"
}
],
"code": 401,
"message": "Login Required"
}
}

So I used "gcloud auth login" and I can use test.bat correctly for now.

@zelon
Copy link
Author

zelon commented Nov 8, 2014

With short form like "gsutil", I still cannot use that test.bat. With full path like "C:\Python27\python.exe E:\Programs\google-cloud-sdk\platform\gsutil\gsutil.py" I can use that test.bat

@mfschwartz
Copy link
Contributor

Hi - I've never been able to get gsutil to work on Windows without fully specified paths - that's how we recommend you use gsutil on Windows (see the Windows tabs of the various pieces of documentation at https://cloud.google.com/storage/docs/gsutil_install).

I'm closing this issue because it's working as documented. I'm glad you were able to get things working.

@paracha3
Copy link

paracha3 commented Mar 2, 2016

Thanks. This has helped me too.

@MartyMacGyver
Copy link

I just stumbled upon this problem myself (I run several gcloud commands to transfer data periodically). In your batch file be sure to preface gcloud or gsutil or such with call, thus:

call gcloud compute copy-files foobarhost:/home/foobar/blah /my/local/folder --zone my-compute-zone
call gcloud compute copy-files foobarhost:/home/foobar/blah /my/local/folder --zone my-compute-zone
...etc...

Using call prevents the batch from exiting as soon as its child batch exits. You can use %ERRORLEVEL% to check the result of the called command (the GCP scripts set this appropriately). See also http://ss64.com/nt/call.html

@jackRoark
Copy link

THAT is the answer! Thank you Marty!

"There always seems to be a way to fix things." - MacGyver

@mfschwartz
Copy link
Contributor

Indeed, thanks Marty, I didn't know about that either.

@juanda2222
Copy link

i have the same exact problem but when i use the full paths, i am using the paths: "C:\Python27\python.exe" "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gsutil.py" cp %myfile.txt% gs://%myBucket%/%folder%/ getting the exception:

ServiceException: 401 Anonymous caller does not have storage.objects.create access to %myBucket%/%folder%/%myfile.txt%

this does not happen when i use the gsutil comand, Any recomendation?

@juanda2222
Copy link

i have the same exact problem but when i use the full paths, i am using the paths: "C:\Python27\python.exe" "C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gsutil.py" cp %myfile.txt% gs://%myBucket%/%folder%/ getting the exception:

ServiceException: 401 Anonymous caller does not have storage.objects.create access to %myBucket%/%folder%/%myfile.txt%

this does not happen when i use the gsutil comand, Any recomendation?

UPDATE:

i manage to work around the problem creating a new process for every operation as child's of the original batch file process using the call comand. therefore the comands inside the batch files now look something like this:

** call cmd /c gsutil cp %file_1% gs://%bucket%/%folder%/ **
** call cmd /c gsutil cp %file_2% gs://%bucket%/%folder%/ **

note that the /c allows the original batch process to wait for the other sub-process so all the operation looks just as it should be

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants