Script that uploads a file to Google Drive works, but when I try to run that script within another I get an HTTPerror

[filifunk]

I'm somewhat of a beginner.

I have a script (morning3.py) that I run that writes and saves a text file and at the end of the script it's supposed to upload that file into google through the api by calling upon another python script called auth.py.

the morning.py file writes the new file but when it tries to upload the file I get this error:

Traceback (most recent call last):
  File "morning3.py", line 108, in <module>
    auth.uploadfile('test{}.txt'.format(datetime.date.today()), 'test{}.txt'.format(datetime.date.today()), 'txt/csv')
  File "/home/pete/google/auth.py", line 78, in uploadfile
    file = service.files().create(body=file_metadata,media_body=media,fields='id').execute()
  File "/home/pete/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/pete/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 871, in execute
    _, body = self.next_chunk(http=http, num_retries=num_retries)
  File "/home/pete/.local/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/pete/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 1054, in next_chunk
    return self._process_response(resp, content)
  File "/home/pete/.local/lib/python3.8/site-packages/googleapiclient/http.py", line 1085, in _process_response
    raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/upload/drive/v3/files?fields=id&alt=json&uploadType=resumable returned "Bad Request">

The weird thing is that I don't try to upload the file by importing auth.py and using the function that uploads the file...and run auth.py separately...it works fine!

This is auth.py, looking at this now I know there is probably ugly code here...but here it is:

from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from googleapiclient.http import MediaFileUpload
import datetime

# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/drive']

def main():
    """Shows basic usage of the Drive v3 API.
    Prints the names and ids of the first 10 files the user has access to.
    """
    creds = None
    # The file token.pickle stores the user's access and refresh tokens, and is
    # created automatically when the authorization flow completes for the first
    # time.
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file(
                'credentials.json', SCOPES)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)

    service = build('drive', 'v3', credentials=creds)

    # Call the Drive v3 API
    results = service.files().list(
        pageSize=10, fields="nextPageToken, files(id, name)").execute()
    items = results.get('files', [])

    if not items:
        print('No files found.')
    else:
        print('Files:')
        for item in items:
            print(u'{0} ({1})'.format(item['name'], item['id']))

def uploadfile(filename,filepath,mimetype):
    creds = None

    
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file(
                'credentials.json', SCOPES)
            creds = flow.run_local_server(port=0)
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)

    service = build('drive', 'v3', credentials=creds)


    file_metadata = {'name': filename,}
    media = MediaFileUpload(filepath,
                        mimetype=mimetype,
                        resumable=True)
    #import pdb
    #pdb.set_trace()
    file = service.files().create(body=file_metadata,media_body=media,fields='id').execute()
    print ('File ID: %s' % file.get('id'))


if __name__ == '__main__':
    main()
    uploadfile('test{}.txt'.format(datetime.date.today()), 'test{}.txt'.format(datetime.date.today()), 'txt/csv')

in morning3.py, I import auth up top and then at the bottom I have this:

auth.uploadfile('test{}.txt'.format(datetime.date.today()), 'test{}.txt'.format(datetime.date.today()), 'txt/csv')

[busunkim96]

Hi @filifunk,

The error you are getting is 400 bad request - which means that something about the content of the request is invalid.

If you're calling out to the same function with the same arguments I would expect a discrepancy in the response. When you run python auth.py you don't see the error?

[filifunk]

@busunkim96 exactly! It works fine when I run python auth.py...the file shows up in my google drive and everything is good in the world. There must be something I don't understand about api's or something...

[busunkim96]

Hmm, maybe the filepaths? Are the file to upload and the two scripts in the same directory?

[filifunk]

Yep, same directory

[busunkim96]

I suggest you try enabling httplib2 logging and comparing the logs from the two runs.

This will print out the content of the HTTP requests and responses so it should be easier to figure out what's happening.

import httplib2
httplib2.debuglevel = 4

[filifunk]

great! thanks @busunkim96 , I will get into it a little bit later and report back

[filifunk]

hmmm...so this is the output:

connect: (www.googleapis.com, 443)
send: b'GET /discovery/v1/apis/drive/v3/rest HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\nuser-agent: Python-httplib2/0.14.0 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Expires: Sun, 26 Jul 2020 15:53:39 GMT
header: Date: Sun, 26 Jul 2020 15:48:39 GMT
header: Cache-Control: public, max-age=300, must-revalidate, no-transform
header: ETag: "-2NioU2H8y8siEzrBOV_qzRI6kQ/rTlLGOr0iXp8xlYsWK1Z8ezzcns"
header: Vary: Origin
header: Vary: X-Origin
header: Content-Type: application/json; charset=UTF-8
header: Content-Encoding: gzip
header: X-Content-Type-Options: nosniff
header: X-Frame-Options: SAMEORIGIN
header: Content-Security-Policy: frame-ancestors 'self'
header: X-XSS-Protection: 1; mode=block
header: Server: GSE
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
header: Transfer-Encoding: chunked
connect: (www.googleapis.com, 443)
send: b'POST /upload/drive/v3/files?fields=id&alt=json&uploadType=resumable HTTP/1.1\r\nHost: www.googleapis.com\r\naccept: application/json\r\naccept-encoding: gzip, deflate\r\nuser-agent: (gzip)\r\nx-goog-api-client: gdcl/1.10.0 gl-python/3.8.2\r\ncontent-type: application/json\r\nx-upload-content-type: txt/csv\r\nx-upload-content-length: 0\r\ncontent-length: 30\r\nauthorization: Bearer  <token>r\r\n\r\n'
send: b'{"name": "test2020-07-26.txt"}'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Type: text/plain; charset=utf-8
header: X-GUploader-UploadID: AAANsUk2IDyqNxsQ20qnti7HA8DPinfnCsATlFsGsCO8OCypNIVbj6EGRQPaQGx8at879DFgZEB5rvLXrJpT2USdiQs
header: Location: https://www.googleapis.com/upload/drive/v3/files?fields=id&alt=json&uploadType=resumable&upload_id=AAANsUk2IDyqNxsQ20qnti7HA8DPinfnCsATlFsGsCO8OCypNIVbj6EGRQPaQGx8at879DFgZEB5rvLXrJpT2USdiQs
header: Vary: Origin
header: Vary: X-Origin
header: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
header: Pragma: no-cache
header: Expires: Mon, 01 Jan 1990 00:00:00 GMT
header: Date: Sun, 26 Jul 2020 15:48:39 GMT
header: Content-Length: 0
header: Server: UploadServer
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
send: b'PUT /upload/drive/v3/files?fields=id&alt=json&uploadType=resumable&upload_id=AAANsUk2IDyqNxsQ20qnti7HA8DPinfnCsATlFsGsCO8OCypNIVbj6EGRQPaQGx8at879DFgZEB5rvLXrJpT2USdiQs HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-range: bytes 0--1/0\r\ncontent-length: 0\r\nauthorization: Bearer  <token>r\r\nuser-agent: Python-httplib2/0.14.0 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
sendIng a read()able
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Content-Type: text/plain; charset=utf-8
header: X-GUploader-UploadID: AAANsUk2IDyqNxsQ20qnti7HA8DPinfnCsATlFsGsCO8OCypNIVbj6EGRQPaQGx8at879DFgZEB5rvLXrJpT2USdiQs
header: Content-Length: 37
header: Date: Sun, 26 Jul 2020 15:48:39 GMT
header: Server: UploadServer
header: Alt-Svc: h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

The thing that jumps out to me is the "must revalidate." Is there anything else I should plug into morning3.py?

All I have in there for the purpose of uploading the file is:

import auth

and

auth.uploadfile('test{}.txt'.format(datetime.date.today()), 'test{}.txt'.format(datetime.date.today()), 'txt/csv')

and I tried it again today and morning3.py gives me the error, but running auth.py uploads

[busunkim96]

You might want to try renaming auth.py to something else. It might be interacting with the auth library's google.auth module in weird ways.

As a side note, make sure to redact sensitive information from logs before you post it to the issue tracker. I've removed the text after "Bearer" because that is an authorization token.

[filifunk]

Unfortunately, I changed the name to sendit.py and still get the same issue. Thanks for redacting! I had no idea!

[parthea]

Hi @filifunk ,

I'm sorry that it took a while to provide a response. I created a file auth.py with the code you provided and then in another file I called auth using the code below and everything worked. I wasn't able to re-produce the problem. I have a PR open (#1126) to improve support for error_details and I'm hoping that, once the fix is merged, you'll get better error detail when you encounter HTTP 400 errors rather than just 'Bad Request'. I'm going to close this issue but if you're still having trouble please re-open this issue with additional information.

import auth
import datetime
import httplib2
import logging

httplib2.debuglevel = 4
logging.basicConfig(level=logging.INFO)

auth.uploadfile('test{}.txt'.format(datetime.date.today()), 'test{}.txt'.format(datetime.date.today()), 'txt/csv')