no logs found while data can be seen in live stream in grafana cloud

[chesha1]

Logs are sent to loki and grafana in cloud, they can be seen in live stream mode, and the labels they brought appear.

However, the logs themself disappear.

The request body is below, the logs are sent to loki by raw http request.

{
	"streams": [
		{
			"stream": {
				"app": "test",
				"namespace": "test2"
			},
			"values": [
				[
					"1715313600000000000",
					"{\"level\":\"info\",\"message\":\"Hello world!\"}"
				],
				[
					"1715313600000000000",
					"{\"level\":\"info\",\"message\":\"Hello world!\"}"
				]
			]
		}
	]
}

They can be seen in live stream:

But they disappeared when querying:

The query time rangeis correct, I use the the last 7 days option.

[paul1r]

The data you sent via the curl command is in the test2 namespace, but your query is in the test3 namespace. Could this be what you are running into?

[chesha1]

The data you sent via the curl command is in the test2 namespace, but your query is in the test3 namespace. Could this be what you are running into?

No, I have tried test, test2, test3, every possible logql sentence finds nothing.

By the way, label selector can find the label shortly after the data was sent. But after a long time, the label selector found nothing either. It seems like logs has some default expiration period.

I don't know whether the above phenomenon show more evidence to find the reason behind it

[paul1r]

May I access your instance?

[chesha1]

May I access your instance?

Sure, but please give me an e-mail address, I'll send account and password to you.
If you do not want to leak e-mail address to public access, you could send an e-mail to chesha1@163.com to let me know.

update:
I am sorry I cannot react quickly in next 10 hours, it is already 1.46 a.m. in GMT+8 and I need to work tomorrow, so I have to sleep now.

[paul1r]

No worries, I work here, just needed your permission to access. :)

I logged in and definitely was able to reproduce what you are seeing. I also have been experimenting with my own Grafana cloud instance.

I wonder if there's a slight error with the curl command?

I was running curl -v -H "Content-Type: application/json" -POST -s "$LOKI_URL/loki/api/v1/push" -d @test.json and was able to see my logs pushed successfully.
Can you post the output of that for yourself?

For reference, my test.json contained:

{
    "streams": [
        {
            "stream": {
                "label3": "value1"
            },
            "values": [
                ["1715369732907106983", "log message 1"]
            ]
        }
    ]
}

[chesha1]

No worries, I work here, just needed your permission to access. :)

I logged in and definitely was able to reproduce what you are seeing. I also have been experimenting with my own Grafana cloud instance.

I wonder if there's a slight error with the curl command?

I was running curl -v -H "Content-Type: application/json" -POST -s "$LOKI_URL/loki/api/v1/push" -d @test.json and was able to see my logs pushed successfully. Can you post the output of that for yourself?

For reference, my test.json contained:

{
    "streams": [
        {
            "stream": {
                "label3": "value1"
            },
            "values": [
                ["1715369732907106983", "log message 1"]
            ]
        }
    ]
}

I have run the same curl as yours, but once again new labels appeared in grafana, but the logs themself cannot be queried.

And the loki url I used is https://787766:glc_eyJvIj---{some characters hidden here}-----0xIn19@logs-prod-020.grafana.net/loki/api/v1/push

Below is the response after curl, it may help find the resaon:

*   Trying 54.251.81.84:443...
* TCP_NODELAY set
* Connected to logs-prod-020.grafana.net (54.251.81.84) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=New York; L=New York; O=Raintank Inc.; CN=grafana.com
*  start date: Feb  2 00:00:00 2024 GMT
*  expire date: Mar  4 23:59:59 2025 GMT
*  subjectAltName: host "logs-prod-020.grafana.net" matched cert's "*.grafana.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Server auth using Basic with user '787766'
> POST /loki/api/v1/push HTTP/1.1
> Host: logs-prod-020.grafana.net
> Authorization: Basic Nzg3NzY2OmdsY19leUp2SWpvaU1UQXpNVGcxTnlJc0ltNGlPaUp6ZEdGamF5MDRNek0zTlRJdGFHd3RkM0pwZEdVdGJHOW5jeUlzSW1zaU9pSTFjamc0T1VaWlMwUXdNelExZDJOa05sRm5NVWxUYjBFaUxDSnRJanA3SW5JaU9pSndjbTlrTFdGd0xYTnZkWFJvWldGemRDMHhJbjE5
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 204
> 
* upload completely sent off: 204 out of 204 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Date: Sat, 11 May 2024 03:14:17 GMT
< 
* Connection #0 to host logs-prod-020.grafana.net left intact
root@server1:~/grafana# ./command.sh 
*   Trying 54.251.81.84:443...
* TCP_NODELAY set
* Connected to logs-prod-020.grafana.net (54.251.81.84) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=New York; L=New York; O=Raintank Inc.; CN=grafana.com
*  start date: Feb  2 00:00:00 2024 GMT
*  expire date: Mar  4 23:59:59 2025 GMT
*  subjectAltName: host "logs-prod-020.grafana.net" matched cert's "*.grafana.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Server auth using Basic with user '787766'
> POST /loki/api/v1/push HTTP/1.1
> Host: logs-prod-020.grafana.net
> Authorization: Basic Nzg3NzY2OmdsY19leUp2SWpvaU1UQXpNVGcxTnlJc0ltNGlPaUp6ZEdGamF5MDRNek0zTlRJdGFHd3RkM0pwZEdVdGJHOW5jeUlzSW1zaU9pSTFjamc0T1VaWlMwUXdNelExZDJOa05sRm5NVWxUYjBFaUxDSnRJanA3SW5JaU9pSndjbTlrTFdGd0xYTnZkWFJvWldGemRDMHhJbjE5
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 204
> 
* upload completely sent off: 204 out of 204 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Date: Sat, 11 May 2024 03:14:33 GMT
< 
* Connection #0 to host logs-prod-020.grafana.net left intact

[paul1r]

I'm seeing logs in your instance now. explore query

[chesha1]

I'm seeing logs in your instance now. explore query

From your link, I can now see the previous logs now. Thank you.

But there's a long delay from sending them to seeing them, is it an expected behaviour?

[paul1r]

I think what you are seeing is a side effect of ingesting "old" logs. This is very nuanced, but if you send logs that are not recent (around "now"), they are not immediately queryable. (There's also nuances about sending items out of order too, but that isn't the issue in this case)

For a curl command, you could try using something like "$(date +%s)"000000000 to put a current timestamp on it.

At this point though, I would recommend playing around with ingesting logs in a traditional fashion. The promtail distribution within Loki is feature-complete and won't receive new updates, but is functional. Alternatively, you could use Grafana Alloy to ingest logs.

Another possible idea would be to use this article as a guide on sending your shell history to Loki.

I'm glad you are up and running, and I hope you find value with Loki!

[paul1r]

It occurred to me that my answer may not have been entirely clear. Generally, when ingesting logs that are "now", you'll see them and they will be queryable almost instantaneously. What you are seeing is a side effect of "old" logs. I hope that helps.

[chesha1]

It occurred to me that my answer may not have been entirely clear. Generally, when ingesting logs that are "now", you'll see them and they will be queryable almost instantaneously. What you are seeing is a side effect of "old" logs. I hope that helps.

Thank you very much for your help. I truly appreciate your kindness in helping me find the appropriate method to send logs to Grafana Cloud using Loki.

When I send logs with the current timestamp, everything works as expected.