Releases: grafana/loki
v2.2.1
2.2.1 fixes several important bugs, it is recommended everyone running 2.2.0 upgrade to 2.2.1
2.2.1 also adds the labelallow pipeline stage in Promtail which lets an allowlist be created for what labels will be sent by Promtail to Loki.
- 3468 adityacs: Support labelallow stage in Promtail
- 3502 cyriltovena: Fixes a bug where unpack would mutate log line.
- 3540 cyriltovena: Support for single step metric query.
- 3550 cyriltovena: Fixes a bug in MatrixStepper when sharding queries.
- 3566 cyriltovena: Properly release the ticker in Loki client.
- 3573 cyriltovena: Fixes a race when using specific tenant and multi-client.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.2.1"
$ docker pull "grafana/promtail:2.2.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.2.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.2.0
🎉🎉 Loki 2.2.0 is released!! 🎉🎉
Please read the Upgrade Guide before upgrading for a smooth experience.
Upgrade Guide TL;DR Loki 2.2 changes the internal chunk format which limits what versions you can downgrade to, a change was made in how many queries were allowed to be scheduled per tenant which might affect your max_query_parallelism and max_outstanding_per_tenant settings, and we fixed a mistake related scrape_configs which do not have a pipeline_stages defined. If you have any Promtail scrape_configs which do not specify pipeline_stages you should go read the upgrade notes!
With over 200 PR's 2.2 includes significant features, performance improvements, and bug fixes!
The most upvoted issue for Loki was closed in this release! Issue 74 requesting support for handling multi-line logs in Promtail was implemented in PR 3024. Thanks @jeschkies!
Other exciting news for Promtail, PR 3246 by @cyriltovena introduces support for reading Windows Events!
Switching to Loki, @owen-d has added a write ahead log to Loki! PR 2981 was the first of many as we have spent the last several months using and abusing our write ahead logs to flush out any bugs!
A significant number of the PR's in this release have gone to improving the features introduced in Loki 2.0. @cyriltovena overhauled the JSON parser in PR 3163 (and a few other PR's), to provide both a faster and smarter parsing to only extract JSON content which is used in the query output. The newest Loki squad member @dannykopping fine tuned the JSON parser options in PR 3280 allowing you to specific individual JSON elements, including support now for accessing elements in an array. Many, many other additional improvements have been made, as well as several fixes to the new LogQL features added some months ago, this upgrade should have everyone seeing improvements in their queries.
@cyriltovena also set his PPROF skills loose on the Loki write path which resulted in about 8x less memory usage on our distributors and a much more stable memory usage when ingesters are flushing a lot of chunks at the same time.
There are many other noteworthy additions and fixes, too many to list, but we should call out one more feature all you Google Cloud Platform users might be excited about: in PR 3083 @kavirajk added support to Promtail for listening on Google Pub/Sub topics, letting you setup log sinks for your GCP logs to be ingested by Promtail and sent to Loki!
For a full list of changes, please checkout the CHANGELOG
Thanks to everyone for another exciting Loki release!!
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.2.0"
$ docker pull "grafana/promtail:2.2.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.2.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.1.0
🎆🎁Happy Holidays from the Loki team!🎁🎆
Please enjoy Loki 2.1.0 to welcome in the New Year!
This release contains a number of fixes, performance improvements and enhancements to the 2.0.0 release!
Notable changes
Helm users read this!
The Helm charts have moved!
- 2720 torstenwalter: Deprecate Charts as they have been moved
This was done to consolidate Grafana's helm charts for all Grafana projects in one place: https://github.com/grafana/helm-charts/
From now moving forward, please use the new Helm repo url: https://grafana.github.io/helm-charts
The charts in the Loki repo will soon be removed so please update your Helm repo to the new URL and submit your PR's over there as well
Special thanks to @torstenwalter, @unguiculus, and @scottrigby for their initiative and amazing work to make this happen!
Also go check out the microservices helm chart contributed by @unguiculus in the new repo!
Fluent bit plugin users read this!
Fluent bit officially supports Loki as an output plugin now! WoooHOOO!
However this created a naming conflict with our existing output plugin (the new native output uses the name loki) so we have renamed our plugin.
- 2974 hedss: fluent-bit: Rename Fluent Bit plugin output name.
In time our plan is to deprecate and eliminate our output plugin in favor of the native Loki support. However until then you can continue using the plugin with the following change:
Old:
[Output]
Name loki
New:
[Output]
Name grafana-loki
Fixes
A lot of work went into 2.0 with a lot of new code and rewrites to existing, this introduced and uncovered some bugs which are fixed in 2.1:
- 2807 cyriltovena: Fix error swallowed in the frontend.
- 2805 cyriltovena: Improve pipeline stages ast errors.
- 2824 owen-d: Fix/validate compactor config
- 2830 sandeepsukhani: fix panic in ingester when not running with boltdb shipper while queriers does
- 2850 owen-d: Only applies entry limits to non-SampleExprs.
- 2855 sandeepsukhani: fix query intervals when running boltdb-shipper in single binary
- 2895 shokada: Fix error 'Unexpected: ("$", "$") while parsing field definition'
- 2902 cyriltovena: Fixes metric query issue with no grouping.
- 2901 cyriltovena: Fixes a panic with the logql.NoopPipeline.
- 2913 cyriltovena: Fixes logql.QueryType.
- 2917 cyriltovena: Fixes race condition in tailer since logql v2.
- 2960 sandeepsukhani: fix table deletion in table client for boltdb-shipper
Enhancements
A number of performance and resource improvements have been made as well!
- 2911 sandeepsukhani: Boltdb shipper query readiness
- 2875 cyriltovena: Labels computation LogQLv2
- 2927 cyriltovena: Improve logql parser allocations.
- 2926 cyriltovena: Cache label strings in ingester to improve memory usage.
- 2931 cyriltovena: Only append tailed entries if needed.
- 2973 cyriltovena: Avoid parsing labels when tailer is sending from a stream.
- 2959 cyriltovena: Improve tailer matcher function.
- 2876 jkellerer: LogQL: Add unwrap bytes() conversion function
Notable mentions
Thanks to @timbyr for adding an often requested feature, the ability to support environment variable expansion in config files!
- 2837 timbyr: Configuration: Support environment expansion in configuration
Thanks to @huikang for adding a new docker-compose file for running Loki as microservices!
- 2740 huikang: Deploy: add docker-compose cluster deployment file
All Changes
For a full list of changes, please checkout the CHANGELOG
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.1.0"
$ docker pull "grafana/promtail:2.1.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.1.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.0.1
2.0.1 is a special release, it only exists to add the v3 support to Loki's chunk format.
There is no reason to upgrade from 2.0.0 to 2.0.1
This chunk version is internal to Loki and not configurable, and in a future version v3 will become the default (Likely 2.2.0).
We are creating this to enable users to roll back from a future release which was writing v3 chunks, back as far as 2.0.0 and still be able to read chunks.
This is mostly a safety measure to help if someone upgrades from 2.0.0 and skips versions to a future version which is writing v3 chunks and they encounter an issue which they would like to roll back. They would be able to then roll back to 2.0.1 and still read v3 chunks.
It should be noted this does not help anyone upgrading from a version older than 2.0.0, that is you should at least upgrade to 2.0.0 before going to a newer version if you are on a version older than 2.0.0.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.0.1"
$ docker pull "grafana/promtail:2.0.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.0.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v2.0.0
🎉🎉 Loki 2.0.0 is here!! 🎉🎉
We are extremely excited about the new features in 2.0.0, unlocking a whole new world of observability of our logs.
Thanks again for the many incredible contributions and improvements from the wonderful Loki community, we are very excited for the future!
Important Notes
Please Note There are several changes in this release which require your attention!
- Anyone using a docker image please go read the upgrade guide!! There is one important consideration around a potentially breaking schema change depending on your configuration.
- MAJOR changes have been made to the boltdb-shipper index, breaking changes are not expected but extra precautions are highly recommended, more details in the upgrade guide.
- The long deprecated
entry_parserconfig in Promtail has been removed, use pipeline_stages instead.
Check the upgrade guide for detailed information on all these changes.
2.0!!!!
There are too many PR's to list individually for the major improvements which we thought justified a 2.0 but here is the high level:
- Significant enhancements to the LogQL query language!
- Generate prometheus alerts directly from your logs!
- Create alerts using the same prometheus alert rule syntax and let Loki send alerts directly to your Prometheus Alertmanager!
- boltdb-shipper is now production ready!
- This is it! Now Loki only needs a single object store (S3,GCS,Filesystem...) to store all the data, no more Cassandra, DynamoDB or Bigtable!
We are extremely excited about these new features, expect some talks, webinars, and blogs where we explain all this new functionality in detail.
Notable mention
This is a small change but very helpful!
- 2737 dlemel8: cmd/loki: add "verify-config" flag
Thank you @dlemel8 for this PR! Now you can start Loki with -verify-config to make sure your config is valid and Loki will exit with a status code 0 if it is!
All Changes
For a full list of changes, please checkout the CHANGELOG
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:2.0.0"
$ docker pull "grafana/promtail:2.0.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v2.0.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v1.6.1
This is a small release and only contains two fixes for Promtail:
- 2542 slim-bean: Promtail: implement shutdown for the no-op server
- 2532 slim-bean: Promtail: Restart the tailer if we fail to read and upate current position
The first only applies if you are running Promtail with both --stdin and --server.disabled=true flags.
The second is a minor rework to how Promtail handles a very specific error when attempting to read the size of a file and failing to do so.
Upgrading Promtail from 1.6.0 to 1.6.1 is only necessary if you have logs full of msg="error getting tail position and/or size",
the code changed in this release has been unchanged for a long time and we suspect very few people are seeing this issue.
No changes to any other components (Loki, Logcli, etc) are included in this release.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:v1.6.1"
$ docker pull "grafana/promtail:v1.6.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.6.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v1.6.0
It's the second thursday of the eighth month of the year which means it's time for another Loki Release!!
Before we highlight important features and changes, congratulations to @adityacs, who is the newest member of the Loki team!
Aditya has been regularly contributing to the Loki project for the past year, with each contribution better than the last.
Many of the items on the following list were thanks to his hard work. Thank you, Aditya, and welcome to the team!
I think we might have set a new record with 189 PR's in this release!
Important Notes
Please Note There are several changes in this release which might require your attention!
- The NET_BIND_SERVICE capability was removed from the Loki process in the docker image, it's no longer possible to run Loki with the supplied image on a port less than 1024
- If you run microservices, there is an important rollout sequence to prevent query errors.
- Scrape configs have changed for Promtail in both Helm and Ksonnet affecting two labels:
instance->podandcontainer_name->container. - Almost all of the Loki Canary metrics were renamed.
- A few command line flags where changed (although they are likely not commonly used)
- If you use ksonnet and run on GCS and Bigtable you may see an error in your config as a default value was removed.
- If you are using boltdb-shipper, you will likekly need to add a new schema_config entry.
Check the upgrade guide for detailed information on all these changes.
Notable Features and Fixes
Query language enhancements
- 2150 introduces
bytes_rate, which calculates the per second byte rate of a log stream, andbytes_over_time, which returns the byte size of a log stream. - 2182 introduces a long list of comparison operators, which will let you write queries like
count_over_time({foo="bar"}[1m]) > 10. Check out the PR for a more detailed description.
Loki performance improvements
- 2216, 2218, and 2219 all improve how memory is allocated and reused for queries.
- 2239 is a huge improvement for certain cases in which a query covers a large number of streams that all overlap in time. Overlapping data is now internally cached while Loki works to sort all the streams into the proper time order.
- 2293 was a big refactor to how Loki internally processes log queries vs. metric queries, creating separate code paths to further optimize metric queries. Metric query performance is now 2 to 10 times faster.
If you are using the query-frontend:
- 2441 improves how label queries can be split and queried in parallel
- 2123 allows queries to the
seriesAPI to be split by time and parallelized; and last but most significant - 1927 allows for a much larger range of queries to be sharded and performed in parallel. Query sharding is a topic in itself, but as a rough summary, this type of sharding is not time dependent and leverages how data is already stored by Loki to be able to split queries up into 16 separate pieces to be queried at the same time.
Promtail
- 2296 allows Promtail to expose the Loki Push API. With this, you can push from any client to Promtail as if it were Loki, and Promtail can then forward those logs to another Promtail or to Loki. There are some good use cases for this with the Loki Docker Logging Driver; if you want an easier way to configure pipelines or expose metrics collection, point your Docker drivers at a Promtail instance.
- 2282 contains an example Amazon Lambda where you can use a fan-in approach and ingestion timestamping in Promtail to work around
out of orderissues with multiple Lambdas processing the same log stream. This is one way to get logs from a high-cardinality source without adding a high-cardinality label. - 2060 introduces the
Replacestage, which lets you find and replace or remove text inside a log line. Combined with 2422 and 2480, you can now find and replace sensitive data in a log line like a password or email address and replace it with ****, or hash the value to prevent readability, while still being able to trace the value through your logs. Last on the list of pipeline additions, - 2496 adds a
Droppipeline stage, which lets you drop log lines based on several criteria options including regex matching content, line length, or the age of the log line. The last two are useful to prevent sending to Loki logs that you know would be rejected based on configured limits in the Loki server.
Logstash output plugin
- 1822 added a Logstash output plugin for Loki. If you have an existing Logstash install, you can now use this plugin to send your logs to Loki to make it easier to try out, or use Loki alongside an existing logging installation.
Loki Canary
- 2344 improved the canaries capabilities for checking for data integrity, including spot checking for logs over a longer time window and running metric queries to verify count_over_time accuracy.
Logcli
- 2470 allows you to color code your log lines based on their stream labels for a nice visual indicator of streams.
- 2497 expands on the series API query to Loki with the
--analyze-labelsflag, which can show you a detailed breakdown of your label key and value combinations. This is very useful for finding improper label usage in Loki or labels with high cardinality. - 2482, in which LogCLI will automatically batch requests to Loki to allow making queries with a
--limit=far larger than the server side limit defined in Loki. LogCLI will dispatch the request in a series of queries configured by the--batch=parameter (which defaults to 1000) until the requested limit is reached!
Misc
- 2453 improves the error messages when a query times out, as
Context Deadline Exceededwasn’t the most intuitive. - 2336 provides two new flags that will print the entire Loki config object at startup. Be warned there are a lot of config options, and many won’t apply to your setup (such as storage configs you aren’t using), but this can be a really useful tool when troubleshooting. Sticking with the theme of best for last,
- 2224 and 2288 improve support for running Loki with a shared Ring using memberlist while not requiring Consul or Etcd. We need to follow up soon with some better documentation or a blog post on this!
All Changes
For a full list of changes, please checkout the CHANGELOG
Dependencies
- Go Version: 1.14.2
- Cortex Version: 7014ff11ed70d9d59ad29d0a95e73999c436c47c
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:1.6.0"
$ docker pull "grafana/promtail:1.6.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.6.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v1.5.0
It's been a busy month and a half since 1.4.0 was released, and a lot of new improvements have been added to Loki since!
Be prepared for some configuration changes that may cause some bumps when upgrading,
we apologize for this but are always striving to reach the right compromise of code simplicity and user/operating experience.
In this case we opted to keep a simplified configuration inline with Cortex rather than a more complicated and error prone internal config mapping or difficult to implement support for multiple config names for the same feature.
This does result in breaking config changes for some configurations, however, these will fail fast and with the list of diffs from the Cortex project should be quick to fix.
Important Notes
Be prepared for breaking config changes. Loki 1.5.0 vendors cortex v1.0.1-0.20200430170006-3462eb63f324,
there were substantial breaking config changes in Cortex 1.0 which standardized config options, and fixed typos.
The Loki docker image user has changed to no longer be root
Check the upgrade guide for more detailed information on these changes.
Notable Features and Fixes
There are quite a few we want to mention listed in order they were merged (mostly)
- 1837 sandeepsukhani: flush boltdb to object store
This is perhaps the most exciting feature of 1.5.0, the first steps in removing a dependency on a separate index store! This feature is still very new and experimental, however, we want this to be the future for Loki. Only requiring just an object store.
If you want to test this new feature, and help us find any bugs, check out the docs to learn more and get started.
- 2073 slim-bean: Loki: Allow configuring query_store_max_look_back_period when running a filesystem store and boltdb-shipper
This is even more experimental than the previous feature mentioned however also pretty exciting for Loki users who use the filesystem storage. We can leverage changes made in 1837 to now allow Loki to run in a clustered mode with individual filesystem stores!
Please check out the last section in the filesystem docs for more details on how this works and how to use it!
- 2095 cyriltovena: Adds backtick for the quoted string token lexer.
This will come as a big win to anyone who is writing complicated reqular expressions in either their Label matchers or Filter Expressions. Starting now you can use the backtick to encapsulate your regex and not have to do any escaping of special characters!!
Examples:
{name="cassandra"} |~ `error=\w+`
{name!~`mysql-\d+`}
- 2055 aknuds1: Chore: Fix spelling of per second in code
This is technically a breaking change for anyone who wrote code to processes the new statistics output in the query result added in 1.4.0, we apologize to anyone in this situation but if we don't fix this kind of error now it will be there forever.
And at the same time we didn't feel it was appropriate to make any major api revision changes for such a new feature and simple change. We are always trying to use our best judgement in cases like this.
- 2031 cyriltovena: Improve protobuf serialization
Thanks @cyriltovena for another big performance improvement in Loki, this time around protbuf's!
- 2021 slim-bean: Loki: refactor validation and improve error messages
- 2012 slim-bean: Loki: Improve logging and add metrics to streams dropped by stream limit
These two changes standardize the metrics used to report when a tenant hits a limit, now all discarded samples should be reported under loki_discarded_samples_total and you no longer need to also reference cortex_discarded_samples_total.
Additionally error messages were improved to help clients take better action when hitting limits.
- 1970 cyriltovena: Allow to aggregate binary operations.
Another nice improvement to the query language which allows queries like this to work now:
sum by (job) (count_over_time({namespace="tns"}[5m] |= "level=error") / count_over_time({namespace="tns"}[5m]))
- 1713 adityacs: Log error message for invalid checksum
In the event something went wrong with a stored chunk, rather than fail the query we ignore the chunk and return the rest.
- 2066 slim-bean: Promtail: metrics stage can also count line bytes
This is a nice extension to a previous feature which let you add a metric to count log lines per stream, you can now count log bytes per stream.
Check out this example to configure this in your promtail pipelines.
- 1935 cyriltovena: Support stdin target via flag instead of automatic detection.
Third times a charm! With 1.4.0 we allowed sending logs directly to promtail via stdin, with 1.4.1 we released a patch for this feature which wasn't detecting stdin correctly on some operating systems.
Unfortunately after a few more bug reports it seems this change caused some more undesired side effects so we decided to not try to autodetect stdin at all, instead now you must pass the --stdin flag if you want Promtail to listen for logs on stdin.
- 2076 cyriltovena: Allows to pass inlined pipeline stages to the docker driver.
- 1906 cyriltovena: Add no-file and keep-file log option for docker driver.
The docker logging driver received a couple very nice updates, it's always been challenging to configure pipeline stages for the docker driver, with the first PR there are now a few easier ways to do this!
In the second PR we added config options to control keeping any log files on the host when using the docker logging driver, allowing you to run with no disk access if you would like, as well as allowing you to control keeping log files available after container restarts.
- 1864 cyriltovena: Sign helm package with GPG.
We now GPG sign helm packages!
All Changes
Too many to list here, see the CHANGELOG for a full list of changes!
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:1.5.0"
$ docker pull "grafana/promtail:1.5.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.5.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v1.4.1
This is release v1.4.1 of Loki.
Important Notes
Please read the Important Notes section for v1.4.0
Mostly be sure to check out the upgrading page, everything for 1.4.0 applies to 1.4.1
Notable changes:
We realized after the release last week that piping data into promtail was not working on Linux or Windows, this should fix this issue for both platforms:
- 1893 cyriltovena: Removes file size check for pipe, not provided by linux.
Also thanks to @dottedmag for providing this fix for Fluent Bit!
- 1890 dottedmag: fluentbit: JSON encoding: avoid base64 encoding of []byte inside other slices
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:1.4.1"
$ docker pull "grafana/promtail:1.4.1"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.4.1/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"v1.4.0
This is release v1.4.0 of Loki.
Over 130 PR's merged for this release, from 40 different contributors!! We continue to be humbled and thankful for the growing community of contributors and users of Loki. Thank you all so much.
Important Notes
Really, this is important
Before we get into new features, version 1.4.0 brings with it the first (that we are aware of) upgrade dependency.
We have created a dedicated page for upgrading Loki in the operations section of the docs
The docker image tag naming was changed, the starting in 1.4.0 docker images no longer have the v prefix: grafana/loki:1.4.0
Also you should be aware we are now pruning old master-xxxxx docker images from docker hub, currently anything older than 90 days is removed. We will never remove released versions of Loki
Notable Features
Please checkout the CHANGELOG for the full list of changes, however here are some of the most notable:
- 1661 cyriltovena: Frontend & Querier query statistics instrumentation.
The API now returns a plethora of stats into the work Loki performed to execute your query, eventually this will be displayed in some form in Grafana to help users better understand how "expensive" their queries are. Our goal here initially was to better instrument the recent work done in v1.3.0 on query parallelization and to better understand the performance of each part of Loki. In the future we are looking at additional ideas to provide feedback to users to tailor their queries for better performance.
This is a long overdue addition to Promtail which can help setup and debug pipelines, with these new features you can do this to feed a single log line into Promtail:
echo -n 'level=debug msg="test log (200)"' | cmd/promtail/promtail -config.file=cmd/promtail/promtail-local-config.yaml --dry-run -log.level=debug 2>&1 | sed 's/^.*stage/stage/g'-log.level=debug 2>&1 | sed 's/^.*stage/stage/g are added to enable debug output, direct the output to stdout, and a sed filter to remove some noise from the log lines.
The stdin functionality also works without --dry-run allowing you to feed any logs into Promtail via stdin and send them to Loki
These two extensions to LogQL now let you execute queries like this:
* `sum(rate({app="foo"}[5m])) * 2`
* `sum(rate({app="foo"}[5m]))/1e6`
- 1678 slim-bean: promtail: metrics pipeline count all log lines
Now you can get per-stream line counts as a metric from promtail, useful for seeing which applications log the most
- metrics:
line_count_total:
config:
action: inc
match_all: true
description: A running counter of all lines with their corresponding
labels
type: CounterThese two configs let you set the max time a chunk can stay in memory in Loki, this is useful to keep memory usage down as well as limit potential loss of data if ingesters crash. Combine this with the query_ingesters_within config and you can have your queriers skip asking the ingesters for data which you know won't still be in memory (older than max_chunk_age).
NOTE Do not set the max_chunk_age too small, the default of 1h is probably a good point for most people. Loki does not perform well when you flush many small chunks (such as when your logs have too much cardinality), setting this lower than 1h risks flushing too many small chunks.
- 1581 slim-bean: Add sleep to canary reconnect on error
This isn't a feature but it's an important fix, this is the second time our canaries have tried to DDOS our Loki clusters so you should update to prevent them from trying to attack you. Aggressive little things these canaries...
- 1840 slim-bean: promtail: Retry 429 rate limit errors from Loki, increase default retry limits
- 1845 wardbekker: throw exceptions on HTTPTooManyRequests and HTTPServerError so FluentD will retry
These two PR's change how 429 HTTP Response codes are handled (Rate Limiting), previously these responses were dropped, now they will be retried for these clients
* Promtail
* Docker logging driver
* Fluent Bit
* Fluentd
This pushes the failure to send logs to two places. First is the retry limits. The defaults in promtail (and thus also the Docker logging driver and Fluent Bit, which share the same underlying code) will retry 429s (and 500s) on an exponential backoff for up to about 8.5 mins on the default configurations. (This can be changed; see the config docs for more info.)
The second place would be the log file itself. At some point, most log files roll based on size or time. Promtail makes an attempt to read a rolled log file but will only try once. If you are very sensitive to lost logs, give yourself really big log files with size-based rolling rules and increase those retry timeouts. This should protect you from Loki server outages or network issues.
Installation:
The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.
Docker container:
$ docker pull "grafana/loki:1.4.0"
$ docker pull "grafana/promtail:1.4.0"Binary
We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:
$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.4.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"