Experiment to reproduce and workaround ingest errors Loki

loki-ingest.md

Reproduce and workaround too_far_behind error

Payload

{
  "streams": [
    {
      "stream": {
          "host": "localhost",
	        "app": "webserver1"
      },
      "values": [
          [ "1715011130000000000", "line1" ],
          [ "1715081130000000000", "line2" ]
      ]
    }
  ]
}

ingesting payload success.

-bash-5.2$ cat ~/ingest2.json -p | http POST http://localhost:3100/loki/api/v1/push
HTTP/1.1 204 No Content
Date: Tue, 07 May 2024 09:56:41 GMT

Query it back

-bash-5.2$ ./cmd/logcli/logcli query '{app="webserver1"}' -q
2024-05-07T13:25:30+02:00 {} line2
2024-05-07T13:25:30+02:00 {} line1

Reproduce too_far_behind: Change payload

Send payload with 1hr older than last timestamp. Notice timestamp change for line1

{
  "streams": [
    {
      "stream": {
          "host": "localhost",
	        "app": "webserver1"
      },
      "values": [
          [ "1715011130000000000", "line1" ],
          [ "1715081130000000000", "line2" ]
      ]
    }
  ]
}

Reproduce too_far_behind: Send the payload

-bash-5.2$ cat ~/ingest2.json -p | http POST http://localhost:3100/loki/api/v1/push
HTTP/1.1 400 Bad Request
Content-Length: 261
Content-Type: text/plain; charset=utf-8
Date: Tue, 07 May 2024 11:26:12 GMT
X-Content-Type-Options: nosniff

entry with timestamp 2024-05-06 15:58:50 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2024-05-07T10:25:30Z',
user 'fake', total ignored: 1 out of 2 for stream: {app="webserver1", host="localhost", service_name="webserver1"}

Query it back to check those log lines are not ingested

-bash-5.2$ ./cmd/logcli/logcli query '{app="webserver1"}' -q
2024-05-07T13:25:30+02:00 {} line2
2024-05-07T13:25:30+02:00 {} line1

Add extra label to ingest those old log entry (line1)

{
  "streams": [
    {
      "stream": {
          "host": "localhost",
	  "app": "webserver1",
	  "type": "re-ingested"
      },
      "values": [
          [ "1715011130000000000", "line1" ],
          [ "1715081130000000000", "line2" ]
      ]
    }
  ]
}

Ingest new payload

-bash-5.2$ cat ~/ingest2.json -p | http POST http://localhost:3100/loki/api/v1/push
HTTP/1.1 204 No Content
Date: Tue, 07 May 2024 11:27:02 GMT

Query it back to see old data is ingested fine.

-bash-5.2$ ./cmd/logcli/logcli query '{app="webserver1"}' -q
2024-05-07T13:25:30+02:00 {}                   line2
2024-05-07T13:25:30+02:00 {}                   line1
2024-05-07T13:25:30+02:00 {type="re-ingested"} line2