About pgbackrest supports the http protocol

[booven]

Please provide the following information when submitting an issue (feature requests or general comments can skip this):

1. pgBackRest version:2.47

2. PostgreSQL version:12

3. Describe the issue:
   Hello, I see that pgbackrest currently only supports https protocol, is there any relevant option for users to choose, or can users release the restrictions by changing the underlying code? If yes, could you please inform us of the location for modification? Thank you very much.

[dwsteele]

is there any relevant option for users to choose

Not currently.

can users release the restrictions by changing the underlying code

It's not just one place, though in concept it would not be that hard to do. Basically, wherever HTTP is used you'd need to remove the tlsClientNewP() constructor and leave the sckClientNew() constructor, see https://github.com/pgbackrest/pgbackrest/blob/main/src/storage/s3/storage.c#L1226 for an example. Might not be that simple, but that's the idea.

You want to tie that into the endpoint for each service so it would behave this way for http and keep the current behavior for http. If you do that for S3, GCS, and Azure you can submit a PR and we'll likely commit it.

[booven]

Hi, I followed your suggestion to remove the tlsClientNewP() from storage.c and keep the sckClientNew() construct, but it still has the following problem: unexpected eof while reading line, it seems to have nothing to do with the http protocol, I don't know what is causing it.

pgbackrest.conf:

[global]
repo1-path=/data
compress-type=zst
compress-level=1
repo1-retention-full-type=count
repo1-retention-full=7
process-max=10
log-level-console=detail
log-level-file=info
start-fast=y
delta=y
resume=y
archive-check=y
log-path=./pgbackrest_backup/logs
archive-timeout=300
lock-path=./pgbackrest_backup/lock

repo1-s3-bucket=100007
repo1-s3-endpoint=http://100.124.255.12:80
repo1-s3-key=5DCEFE69034761D9A86106DBEBD521F9D
repo1-s3-key-secret=QHTajSR6LM2vzc3
repo1-s3-region=cn-1
repo1-storage-verify-tls=n
repo1-s3-uri-style=path
repo1-type=s3
repo1-block=y
repo1-bundle=y
backup-standby=n

tls-server-address=*
tls-server-cert-file=/app/pg/pg_9501/pgbackrest_backup/certs/backup-srv.crt
tls-server-key-file=/app/pg/pg_9501/pgbackrest_backup/certs/backup-srv.key
tls-server-ca-file=/app/pg/pg_9501/pgbackrest_backup/certs/ca.crt

tls-server-auth=BACKUP_HOST=*

error.txt

Thank you very much！

[dwsteele]

I don't know, either, especially since you did not show your code change.

[booven]

Sorry for forgetting to provide important information, we have made the following changes to pgbackrest:
In src\storage\s3\storage.c：
beafore：
this->;httpClient = httpClientNew( tlsClientNewP( sckClientNew(host, port, timeout, timeout), host, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath), timeout);
after：
this->;httpClient = httpClientNew( sckClientNew(host, port, timeout, timeout), timeout);
before:
this->credHttpClient = httpClientNew( tlsClientNewP( sckClientNew(this->credHost, S3_STS_PORT, timeout, timeout), this->credHost, timeout, timeout, true, .caFile = caFile, .caPath = caPath), timeout);
after:
this->credHttpClient = httpClientNew( sckClientNew(this->credHost, S3_STS_PORT, timeout, timeout), timeout);
In In src\common\io\http\url.c, we have commented out the following
if (param.type != httpProtocolTypeAny && this->pub.type != param.type) THROW_FMT(FormatError, "expected protocol '%s' in URL '%s'", strZ(httpProtocolTypeStr(param.type)), strZ(url));

We use tcpdump to view the http result and find the following error:
`=.@.@.D>...'...,"..C3.D..a..P..X....HTTP/1.1 499 status code 499
Accept-Ranges: bytes
Content-Length: 536
Content-Type: application/xml
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: 6f651f6229893d69d106cc6b54dd3ac474473f96c3c0430f7b392b5672e38d8d
X-Amz-Request-Id: 17CD3794D2206ADE
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Date: Tue, 07 May 2024 13:22:25 GMT
Connection: close

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>ClientDisconnected</Code><Message>Client disconnected before response was ready</Message><Key>data/backup/3ef9f2ab67ec482fb6f20cc619fac613_1/20240507-211746F/bundle/2</Key><BucketName>3ef9f2ab67ec482fb6f20cc619fac613</BucketName><Resource>/3ef9f2ab67ec482fb6f20cc619fac613/data/backup/3ef9f2ab67ec482fb6f20cc619fac613_1/20240507-211746F/bundle/2</Resource><RequestId>17CD3794D2206ADE</RequestId><HostId>6f651f6229893d69d106cc6b54dd3ac474473f96c3c0430f7b392b5672e38d8d</HostId></Error>
21:22:25.455479 IP 192.168.26.44.38469 > 192.168.26.39.8901: Flags [.], seq 2865619878:2865627178, ack 2382115290, win 58, length 7300
E.....@.@......,...'.E".......1.P..:.B..PUT /3ef9f2ab67ec482fb6f20cc619fac613/data/backup/3ef9f2ab67ec482fb6f20cc619fac613_1/20240507-211746F/bundle/1?partNumber=1&uploadId=YmM2Yzg0ZmQtZmUyOC00OWU3LWE0YzMtZDI5ZmZlMDdmZTk5LjcxZWJhYWYwLTY1ZDEtNDAxMy05NDE1LWZjZmRhYTY0NzQ2Yg HTTP/1.1
user-agent:pgBackRest/2.47
authorization:AWS4-HMAC-SHA256 Credential=IMoGRpI9C8SNYoVOnF52/20240507/cn-guangdong-1/s3/aws4_request,SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-date,Signature=f5f65eeaf694c18dc90f59f5350b7f5a65cb36bfd5d5d1954c37365dcdadbcac
content-length:5242880
content-md5:TViNu3uyMaUKcYD8ttM2vA==
host:192.168.26.39
x-amz-content-sha256:79d32f821fcf0c748b8176d7e30068dd1345b44393775aa5fb3b2b4d676ac719
x-amz-date:20240507T132216Z`

And tried to set the io-timeout to the maximum, still no result, it is not clear why.

Looking forward to your reply very much！

[dwsteele]

Yeah, not sure why that is happening, either. This is not something we have ever tested so could be anything.

[booven]

Hello, I have located the specific code as:
this->;httpClient = httpClientNew( tlsClientNewP( sckClientNew(host, port, timeout, timeout), host, timeout, timeout, verifyPeer, .caFile = caFile, .caPath = caPath), timeout);
May I ask whether the removal of TLS encapsulation will lead to early shutdown of the IO client? Currently, it seems that the client was closed before sending was completed, resulting in a put failure。

3ef9f2ab67ec482fb6f20cc619fac613_2_pgbackrest.txt

[dwsteele]

You keep asking the same question and the answer remains the same -- I don't know. If you are getting as far as copying files then it must work to some extent but it seems to choke when it gets to a larger file.

To track down the issue will require debugging and probably writing some new tests -- neither of which I have time to do right now.