Q: Is there a way to query Prometheus to count failed jobs in time range? There are several metrics collected for cron jobs, unfortunately I'm not sure how to use them properly.
I wanted to use the kube_job_status_failed == 1 metrics. I can use a regex for job=~"+.myjobname.+" to aggregate all failed attempts for a cron job.
This is where i got stuck. Is there a way to count the amount of distinct labels(=number of failed attempts) in a given time period?
Or can I use the metrics the other way around meaning checking whether there was kube_job_status_succeeded{job=~"+.myjobname+."}==1 in a given time period?
I feel like I'm so close to solving this but I just can't wrap my head around it.
EDIT: Added PictureThis shows that there clearly are several succeded jobs over time, I just have no clue on how to count them

A: This should give you the number of failed jobs matching the job name in 1h period:
count_over_time(kube_job_status_failed{job=~"+.myjobname+."}==1 [1h])


A: I searched for this answer myself and found offset working for my purpose.
kube_job_failed{job_name=~"^your_job_name.*", namespace="your_teamspace",} - kube_job_failed{job_name=~"^your_job_name.*", namespace="your_teamspace",} offset 6h > 2

I needed 6h, not 1h and the amount of failed jobs to be larger than 2 in this timerange.
