Any recommendations for #Prometheus long term metric storage?
I tweaked Prometheus to store a year's worth of data, but one unclean shutdown and I've got a corrupt WAL block, which it doesn't handle well (needs 20GiB RAM at startup).
There are many options (#OpenTSDB, #Graphite, #InfluxDB, ...), any reasonable low-maintenance / low-resource storage which I could write from Prometheus and show in #Grafana dashboards?
@michal #Thanos uses S3 for persistence. #VictoriaMetrics uses local block storage, but has replication options at better performance than vanilla #prometheus .
@claus I'd like to keep everything local, I'm certainly not going to pay to store some cpu load and temperature/humidity data :)
@albi https://github.com/timescale/promscale/issues/1836 "We regret to inform you that we’ve made the difficult decision to discontinue Promscale" :(
In the end I settled on Grafana Mimir: https://github.com/grafana/mimir
Deployment was easy (reasonable looking config file + Docker arm64 image), local storage to a single directory, reasonable migration story (tool which straight uploaded Prometheus data blocks, backfilled a year's worth of data), Grafana support as data source (duh!).
The only nuisance was that I had to modify every single graph to switch to the new data source...