AWS SageMaker - How to fix "The request signature we calculated does not match the signature you provided"

[naveen-marthala]

I have a sagemaker endpoint, which works as expected from boto3's sagemaker-session's invoke-endpoint, aws-cli. And to invoke it from my app, I have created an user with only permission to invoke this specific endpoint and it worked as expected from Postman, when I have specified credentials in Authorisation tab. And when I try to invoke it with requests(python's builting package for http requests), I get signature matching error. Please help me fix it.

Here's how I have tried to send http post request to AWS SageMaker to invoke my endpoint, with help from this aws-blog:

from datetime import datetime
import hashlib
import hmac
import requests

method = "POST"
service = "sagemaker"
host = "runtime.sagemaker.ap-south-1.amazonaws.com"
region = "ap-south-1"
endpoint = "https://runtime.sagemaker.ap-south-1.amazonaws.com/endpoints/<my-end-point>/invocations"
content_type = "application/json"

# I have hidden my credentials here.
access_key = "<my-key>"
secret_key = "<my-key>"

# Create a date for headers and the credential string
t = datetime.utcnow()
amz_date = t.strftime("%Y%m%dT%H%M%SZ")
date_stamp = t.strftime("%Y%m%d")  # Date w/o time, used in credential scope

canonical_uri = "/"

request_parameters = "{\n \"longitude\": [-121.31, -122.37, -120.43, -119.81, -119.82],\n \"latitude\": [36.42, 37.93, 34.97, 36.83, 36.76],    \n \"housing_median_age\": [121.0, 37.0, 28.0, 19.0, 46.0],\n \"total_rooms\": [12740.0, 709.0, 1433.0, 6789.0, 2194.0],\n \"total_bedrooms\": [1615.0, 190.0, 270.0, 1200.0, 563.0],\n \"population\": [12630.0, 644.0, 1001.0, 2325.0, 924.0],\n \"households\": [1564.0, 174.0, 278.0, 1109.0, 542.0],\n \"median_income\": [12.6629, 0.8641, 4.0125, 4.049, 1.4028]\n }"

canonical_querystring = ""

# Step 4: Create the canonical headers. Header names must be trimmed
canonical_headers = f"content-type:{content_type}\nhost:{host}\nx-amz-date:{amz_date}"

# Step 5: Create the list of signed headers. This lists the headers
# in the canonical_headers list, delimited with ";" and in alpha order.
# Note: The request can include any headers; canonical_headers and
# signed_headers include those that you want to be included in the
# hash of the request. "Host" and "x-amz-date" are always required.
signed_headers = "host;x-amz-content-sha256;x-amz-date"


def sign(key, msg):
    return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()


def getSignatureKey(key, date_stamp, regionName, serviceName):
    kDate = sign(("AWS4" + key).encode("utf-8"), date_stamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, "aws4_request")
    return kSigning


# Step 6: Create payload hash. In this example, the payload (body of
# the request) contains the request parameters.
payload_hash = hashlib.sha256(request_parameters.encode('utf-8')).hexdigest()

# Step 7: Combine elements to create canonical request
# canonical_request = method + '\n' + canonical_uri + '\n' + canonical_querystring + '\n' + canonical_headers + '\n' + signed_headers + '\n' + payload_hash
canonical_request = f"{method}\n{canonical_uri}\n{canonical_querystring}\n{canonical_headers}\n{signed_headers}\n{payload_hash}"

# ************* TASK 2: CREATE THE STRING TO SIGN*************
algorithm = "AWS4-HMAC-SHA256"
credential_scope = f"{date_stamp}/{region}/{service}/aws4_request"

# string_to_sign = algorithm + '\n' +  amz_date + '\n' +  credential_scope + '\n' +  hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()
string_to_sign = f"{algorithm}\n{amz_date}\n{credential_scope}\n{hashlib.sha256(canonical_request.encode('utf-8')).hexdigest()}"


# ************* TASK 3: CALCULATE THE SIGNATURE *************
# Create the signing key using the function defined above.
signing_key = getSignatureKey(secret_key, date_stamp, region, service)

# Sign the string_to_sign using the signing_key
signature = hmac.new(signing_key, string_to_sign.encode('utf-8'), hashlib.sha256).hexdigest()


# ************* TASK 4: ADD SIGNING INFORMATION TO THE REQUEST *************
authorization_header = algorithm + " " + "Credential=" + access_key + "/" + credential_scope + ", " + "SignedHeaders=" + signed_headers + ", " + "Signature=" + signature

headers = {
           "X-Amz-Date": amz_date,
           "Authorization": authorization_header
           }

# ************* SEND THE REQUEST *************
print('\nBEGIN REQUEST++++++++++++++++++++++++++++++++++++')
print('Request URL = ' + endpoint)

# r = requests.post(endpoint, json=request_parameters, headers=headers)
r = requests.request(method=method, url=endpoint, headers=headers, json=request_parameters)

print('\nRESPONSE++++++++++++++++++++++++++++++++++++')
print('Response code: %d\n' % r.status_code)
print(r.text)

and this got printed, throwing me the signature mismatch error.

BEGIN REQUEST++++++++++++++++++++++++++++++++++++
Request URL = https://runtime.sagemaker.ap-south-1.amazonaws.com/endpoints/<my-end-point>/invocations

RESPONSE++++++++++++++++++++++++++++++++++++
Response code: 403

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/endpoints/<my-end-point>/invocations\n\nhost:runtime.sagemaker.ap-south-1.amazonaws.com\nx-amz-content-sha256:\nx-amz-date:20210901T182952Z\n\nhost;x-amz-content-sha256;x-amz-date\n4ea75b8f9815505ffbf7ee43eb75b9496cc0641b2ae13efe4d19b1f89ba27751'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20210901T182952Z\n20210901/ap-south-1/sagemaker/aws4_request\n55ab4ae142990e512b0a1c10646b1e7a1be583396f838bbbc543380b155639be'\n"}

So, What have I missed and how do I fix this signature mis-match error.
And, I can't use boto3 or any additional python library apart from the builtin python libraries available with a Debian 10 installation.

[kamtingtsoi]

Hi, this thread is over 2 years old, but I have recently come across the same issue as yours and figured out the problem was an incorrect canonical_uri.

In your case, the canonical_uri is "/endpoints/<my-end-point>/invocations"
In fact, you could derive the host and uri using the urllib library, as below:

from urllib.parse import urlparse

host = urlparse(endpoint).hostname
canonical_uri = urlparse(endpoint).path or "/"

Hope this helps!