Currently, certs for alchemyof.it cannot be requested when cloudflare is configured in strict TLS mode. Investigate adding cloudflare cert issuer to cert-manager or reconfiguring cloudflare rules.

Alert TargetDown firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2022-12-22 13:39:48.970753442 +0000 UTC m=+327629.299195271.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: c3058ff715a6bb277bc9d4d65713d6bde6d43ea2e64facf2cee7f953c750f083 )

• argoCD - use following images for multi-arch support: https://hub.docker.com/r/phillebaba/argocd custom build images from https://quay.io/repository/paulfantom/argocd
• ZNC IRC bouncer moved to riot.im
• calibre-web
• guacamole, based on helm chart
• homer or heimdall
• ssh bastion host manager - bastillion not needed anymore
• tekton CI no need as github actions used most of the time
• keycloak as OIDC backend (needs investigation) or maybe use nextcloud?
• nextcloud-exporter (added in #24)
• move blog to ghost and host on cluster
• add shynet for blog data collection and analysis not necessary anymore
• grafana-operator grafana in separate namespace
• unifi-poller instead of current unifi-exporter
• promtail + loki
• pushgateway at http://push.monitoring.svc:9091
• photoprism

Non-containerized:

• earlyoom

In addition to NFS provisioner it would be beneficial to setup iSCSI provisioner - https://github.com/kubernetes-incubator/external-storage/tree/master/iscsi/targetd

Majority of applications using hostPath for volumes could be moved to use iSCSI PVC.

Action items:

• Ansible setup for targetd server (on NAS)
• Ansible setup for iSCSI initiators (everywhere)
• Manifests for iSCSI provisioner
• 2 storage classes - one for vg_fast and one for vg_storage. A former one could be used for testing. Only vg_fast is allowed to be accessible for kubernetes. vg_storage is full and allowed to be accessed only as hostPath for performance reasons.

K3S offers a way to customize coredns responsible for cluster DNS setup. It would be nice to use it for a split-DNS situation.

Lead - k3s-io/k3s#4397

(Updated at 2022-12-18 17:00:29.598843205 +0000 UTC m=+588.227325790)

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 933b432e797b2d35572313028cb4a685383488c087c4aea733e81833860129d5 )

Unifi controller creates backup every week. However, this backup is stored locally on controller itself. It would be beneficial to create a cronjob to copy backup files and store on nfs-backed PV.
Backup files are stored in /data/autobackup/ on unifi controller.

Next step would be to add backup job to send data from PV to Backblaze.

Alert KubeDaemonSetRolloutStuck firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2024-02-10 11:32:40.199340918 +0000 UTC m=+44472.207561420.

Common Labels

Common Annotations

Alerts

Currently, there are many manually deployed services on NAS. This needs to be adjusted and config should be managed via git.

Discovered services:

• ddclient
• iscsi targetd not installed, issue tracked in #2
• NFS exports

Configuration done on install (may be possible to automate via kickstart):

• teamd config for LACP teamed network devices done during system installation
• mdadm config done during installation
• LVM config done during installation
• Static DNS configuration (main DNS server for other clients is running in cluster) done during installation

Alert KubeDaemonSetMisScheduled firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2022-12-22 13:39:43.58186847 +0000 UTC m=+327623.910310279.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 7d04718e7ad227b0d6c6089159e34b440211567712f5bb11c4fc412328114d13 )

Alert KubePodCrashLooping firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2024-04-02 20:41:58.294093185 +0000 UTC m=+20.802279370.

Common Labels

Common Annotations

Alerts

Alert PostgreSQLHighConnections firing in paperless namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-04-25 20:19:26.626114387 +0000 UTC m=+82818.323127310.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: d5f17e654e53aba5a290a7b5bd6083f74ceac73c01297e6d6edaf7b22b53a627 )

Alert TestAlert firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2022-12-18 18:39:46.271423238 +0000 UTC m=+26.599865047.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 346585aa11eea0e4b4d76f6e02fbcb6fd3e072e0044f467030c7eaa1440195b6 )

Main point behind migrating to postgresql:

• it is faster (source)
• mysql doesn't have good monitoring coverage in prometheus ecosystem (lack of meaningful alerts). Postgresql, on the other hand, is used by gitlab and has already established alerts and runbooks (alerts, recording rules)
• possibly no hacks required to run postgresql_exporter

Cons:

• there might be issues with some nextcloud apps not working with postgre as backend (more in nextcloud/server#5912 (comment) and nextcloud/twofactor_admin#35)

Alert TargetDown firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2024-04-14 20:39:22.674050872 +0000 UTC m=+214945.203274758.

Common Labels

Common Annotations

Alerts

Alert PostgreSQLCacheHitRatio firing in homeassistant namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-01-17 13:36:14.31084953 +0000 UTC m=+370286.411825553.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: b0cb3dbb66ae7f9be3e3a19466b1c79090cff7825d7e5292013a2f97e4dc16e4 )

Alert TargetDown firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-03-07 15:36:04.518855456 +0000 UTC m=+5316.354817277.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 450f142a6083f4ed63d0d1d7de01ebeb9d33d15312095933d5f927bb4e076c3f )

Due to resource constraints on master01, ansible deployment cannot finish and causes k3s apiserver to crash. To mitigate issue it would be better to run ansible as a cronjob in k3s.

DoD:

• find/create a container image with:
  • ansible
  • ssh client
  • git
• crojob should run ./deploy.sh script
• repository should be mounted as PV on NFS storage class
• pushgateway shouldn't be exposed to local network anymore
• ansible_connection=local cannot be set for any host

docs/ directory could be hosted externally under docs.ankhmorpork.thaum.xyz. This can be done by using pattern from https://github.com/khuedoan/homelab/tree/master/docs

• Reduce severity level of SiteDown alert to warning
• Recreate SiteExternallyDown alert that uses only uptimerobot data
• Introduce inhibition rule - if SiteExternallyDown is firing, don't send SiteDown

Checkout if https://github.com/benbjohnson/litestream can be used to increase the robustness of applications using SQLite underneath

Alertmanager URL: http://localhost:9093

• firing

  Labels:

  • thing = value

  Annotations:

  • hint = how to fix foobar

TODO: add graph url from annotations.

organizr v2 might be a better fit for main portal site.

Investigate why UDM Pro did not update DNS entry.

Alertmanager URL: http://localhost:9093

• firing

  Labels:

  • alertname = Test
  • namespace = monitoring
  • thing = value

  Annotations:

  • hint = how to fix foobar

TODO: add graph url from annotations.

Alert TargetDown firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-03-07 14:39:19.103654196 +0000 UTC m=+1910.939616008.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 0f273fe6719899752f139b038340c40a5684ab5eda17b89da23492695f528c41 )

Investigate k8s-native backup solutions to swap the current custom backup solution to a generic one.

Requirements:

• uses restic internally
• allow sending backups to Backblaze
• backups need to be encrypted
• allow backing up any PV

Nice to have:

• allow backing up /var/lib/rancher/k3s/server on master node to allow master node recovery

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update base infrastructure (devsec.hardening, k3s-io/k3s, prometheus.prometheus)
• chore(deps): update dependency golang to v1.22.2

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Alert PostgreSQLCacheHitRatio firing in namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2022-12-19 02:13:13.971634309 +0000 UTC m=+27234.300076138.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 5ef6b1689008f5f35e66c52c08ad18997835c6f7116a07f925d42ce9d44bf43c )

Redis cache was disabled after a recent outage and disaster recovery. It needs to be reenabled and improved by adding password protection (REDIS_PASSWORD variable in nextcloud).

Unfortunately due to how nextcloud docker container is constructed, this is a manual work on nextcloud side and needs to be done by editing config.php file as well as setting correct variables for nextcloud container.

I am using Prometheus Probe CRD and Blackbox exporter to scrape static targets. But, when I checked in Blackbox exporter, I don't see specified targets being probed at all.

I was able to probe targets using Blackbox exporter and additionalScrapeConfigs in values file of Prometheus exporter but it doesn't work with Probe CRD.

Here is my Probe custom object config,

kind: Probe
metadata:
  name: probe-crd
  namespace: prometheus
spec:
  jobName: probe-crd
  prober:
    url: prometheus-blackbox-exporter:9115
  targets:
    staticConfig:
      static:
      - https://www.google.com

Blackbox exporter service is running on port 9115. Can someone please let me know what I am missing here?

Alert ThanosQueryGrpcClientErrorRate firing in datalake-metrics namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2024-03-22 02:21:45.539869555 +0000 UTC m=+24.538879833.

Common Labels

Common Annotations

Alerts

List of collections of alerts/recording rules and runbooks which might be useful for the cluster:

• https://gitlab.com/gitlab-com/runbooks/-/tree/master
• https://github.com/samber/awesome-prometheus-alerts doesn't have anything more meaningful than what is already sourced from other projects

Tools to run in CI:

• https://github.com/FairwindsOps/pluto - added in #16
• kubeval - added in 011b7b2

Alert PostgreSQLMaxConnectionsReached firing in paperless namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-04-28 15:13:57.585697289 +0000 UTC m=+22567.696417215.

Common Labels

Common Annotations

Alerts

(DO NOT MODIFY: 9af083c4c04aefaa4b687f93fe36767e0219bc14a37a0deb4214a22c8f6ab837 )

Alert ThanosStoreObjstoreOperationLatencyHigh firing in datalake-metrics namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-12-04 05:27:59.510352973 +0000 UTC m=+495.481901453.

Common Labels

Common Annotations

Alerts

Alert ReconciliationFailure firing in flux-system namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2024-04-15 17:13:33.601854197 +0000 UTC m=+288996.131078121.

Common Labels

Common Annotations

Alerts

Currently, the used version is outdated and needs to be updated to https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner

Seems like permissions for ConfigMapSecret controller are too broad as discussed in https://kubernetes.slack.com/archives/CFFDS2Z7F/p1594757903351300

• Enable BGP in Unifi USG
• Switch metallb to use BGP-mode instead of ARP-mode

Instructions: https://community.ui.com/questions/BGP-instructions-for-USG-K8s-MetalLB/b61e2f67-34f2-4571-9140-8d6b9cde2d72

• cert-manager
• metallb (use metallb/metallb#147 as base)
• ingress-nginx
• coreDNS (use https://github.com/povilasv/coredns-mixin), added in 1bb0047 after some configuration
• flux v2
• harbor
• oauth2-proxy

It was disabled in 92b4e47

Recent code update broke down DNS and adblocker plugin cannot contact external sources to pull blocklists.

Example errors:

[WARNING] plugin/ads: Loading list from url "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" failed with error: Get "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts": net/http: TLS handshake timeout
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 www.google.de. A: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[WARNING] plugin/ads: Loading list from url "https://mirror1.malwaredomains.com/files/justdomains" failed with error: Get "https://mirror1.malwaredomains.com/files/justdomains": dial tcp 139.146.167.17:443: i/o timeout
[ERROR] plugin/errors: 2 imap.gmail.com. AAAA: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out
[ERROR] plugin/errors: 2 . NS: tls: DialWithDialer timed out

Alert Test firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2022-12-18 18:18:19.43869783 +0000 UTC m=+24.267089034.

Common Labels

Common Annotations

<td>some description</td>

<td><a href="https://runbooks.thaum.xyz">https://runbooks.thaum.xyz</a></td>

Alerts

(DO NOT MODIFY: 21a57638781bc59a1dbd9cc17208d8dd0867816312710c3eaf12f759182010ee )

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: Invalid JSON (parsing failed)
Message: Syntax error: expecting String near ntom"], }

DIUN detected new version of testimage available. Consider upgrading.

Since CoreDNS is already used in k3s cluster, it might be a good idea to replace pi-hole with CoreDNS.

Pros:

• one server type to learn
• native prometheus metrics exposition (instead of faulty eko/pihole-exporter) with proper metrics :)
• DNS-over-HTTP support
• full GitOps management
• stateless and easily scalable

Cons:

• No webUI to quickly whitelist sites
• ads plugin is not included natively in CoreDNS

Alert KubeDeploymentReplicasMismatch firing in monitoring namespace

This is an automated issue created by the monitoring system. Please do not edit this message.

Alertmanager URL: https://alertmanager.ankhmorpork.thaum.xyz

Issue was last updated at 2023-12-06 18:30:40.223380231 +0000 UTC m=+1278.197167614.

Common Labels

Common Annotations

Alerts

Deploy a knowledgebase application like bookstack at wiki.thaum.xyz or wiki.ankhmorpork.thaum.xyz

Alternative would be to use some sort of SaaS offering or anything from https://github.com/awesome-selfhosted/awesome-selfhosted#wikis