https://github.com/google/alertmanager-irc-relay

Probably have alerts go to a channel for that purpose (e.g. #!infra ? )

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Location: renovate.json
Error type: Invalid JSON (parsing failed)
Message: Syntax error near ],

There's an official DigitalOcean operator that can create and manage DO's hosted databases as normal k8s resources.

I think it would be useful to pivot to a userdb instance managed this way, and replace the manually-managed credentials and secrets with automatically provisioned ones.

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• userdb-api: update postgrest/postgrest:v12.0.2 Docker digest to 79369c0
• argocd: update redis Docker tag to v7.2.4
• ingress-nginx/controller: update registry.k8s.io/ingress-nginx/controller Docker tag to v1.10.1
• keycloak: update quay.io/keycloak/keycloak Docker tag to v24.0.3
• wkd: update kiwigrid/k8s-sidecar Docker tag to v1.26.1
• argocd: update ghcr.io/dexidp/dex Docker tag to v2.39.1
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Message templates are here: https://github.com/hashbang/gitops/blob/master/argocd/argocd-notifications/cm-patch.yaml#L22-L29

https://docs.cilium.io/en/stable/policy/

I noticed that this repo uses Argo CD, and I thought I’d let you know that I recently released a Github app called Infro that several companies use internally that allows Argo CD users to preview Kubernetes changes in Github pull requests before they merge. I’m providing it for free to open source projects (here’s an example in the wild). Here’s a setup guide with links to documentation. It’s in early stages, so I’m sure there will be warts. All feedback is welcome!

Many of our resources only select a particular image tag; rather than an exact hash.

• Use kustomization image field to hashlock
• Check any operators for additional images they may bring in

kubectl get pods --all-namespaces -o json | jq '.items[].spec.containers[].image' | grep -v sha256 | sort -u 

• digitalocean/do-csi-plugin:v4.2.0
• docker.io/cilium/cilium:v1.10.4
• docker.io/cilium/operator:v1.10.4
• docker.io/coredns/coredns:1.8.4
• docker.io/digitalocean/arp-flusher:v0.0.2
• docker.io/digitalocean/do-agent:3.11.0
• docker.io/digitalocean/do-csi-plugin:v4.4.1
• hashbang/hashbangctl
• k8s.gcr.io/sig-storage/csi-attacher:v3.5.0
• k8s.gcr.io/sig-storage/csi-provisioner:v3.2.1
• k8s.gcr.io/sig-storage/csi-resizer:v1.5.0
• k8s.gcr.io/sig-storage/csi-snapshotter:v6.0.1
• nginx:1.21.0
• quay.io/jetstack/cert-manager-cainjector:v1.11.2
• quay.io/jetstack/cert-manager-controller:v1.11.2
• quay.io/jetstack/cert-manager-webhook:v1.11.2
• registry.k8s.io/kube-proxy:v1.24.12
• registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.6.0
• 42wim/matterbridge:1.26.0
• drgrove/mtls-server:v0.20.0
• drgrove/wkd:v2.2.2
• eu.gcr.io/k8s-artifacts-prod/external-dns/external-dns:v0.13.4
• ghcr.io/dexidp/dex:v2.36.0-distroless
• ghcr.io/ergochat/ergo:v2.11.1
• hashbang/book:latest
• hashbang/hashbang.sh:latest
• hashbang/webirc:latest
• k8s.gcr.io/sig-storage/csi-node-driver-registrar
• k8s.gcr.io/sig-storage/snapshot-controller
• k8s.gcr.io/sig-storage/snapshot-validation-webhook
• kiwigrid/k8s-sidecar:1.24.0
• postgrest/postgrest:v11.0.1
• quay.io/argoproj/argocd:v2.7.2
• redis:7.0.11-alpine
• redis:7.0.5-alpine
• registry.k8s.io/ingress-nginx/controller:v1.7.1
• thatonecalculator/calckey:v13.1.4.1

Thoughts for consideration:

• Should we use hashbang.sh for this or irc.hashbang.sh?
  • If we use irc.hashbang.sh for this, how can we set up hashbang.sh in a way where IRC won't lose out on messages created on a potential hashbang.sh matrix server
  • If we don't use irc.hashbang.sh, we need a good way to set up either SRV records or well-known, with the former being defined in admin-tools and the latter probably being done via an Ingress. I believe we can have separate Ingress on the same domain name defined across namespaces, so this should be a valid option
• Should this be tested to see if we can use a temporary database?
• Should we use Dendrite in monolith or polylith mode? Polylith mode probably won't give us any advantage, and given this only needs to scale to the size of the IRC server, I think the IRC server will fall over first.

affected applications:

• ingress
• cert-manager

At the project level we can enforce signed gpg commits.

Or an S3 compatible API, if|when we move out of DigitalOcean.

This is useful for us due to free ingress into the system. $5/250GB per month. This is actually cheaper than the current solution using a PVC as those are dynamically generated and are $10/100GB. This also means that we can store more logs over time, and - if we want - we can apply lifecycle rules with DigitalOcean Spaces Block Storage.

As mentioned in #4

We should have a sidecar that watches for configmap changes and sends a SIGHUP or rehash command to the irc server.

Can be migrated from https://github.com/hashbang/infra/tree/master/charts/site

Requires kubectl version 1.18.8+ (about, I didn't do thorough testing, but definitely not 1.18.3).

https://www.digitalocean.com/docs/kubernetes/how-to/monitor-advanced/

The digital ocean agents are failing to connect to kube-state-metrics; I think kube-rbac-proxy is the gatekeeper.

I've been playing with the kube-system daemonset.apps/do-node-agent argument --k8s-metrics-path=https://kube-state-metrics.monitoring.svc.cluster.local:8443/metrics to try and fix it.

https://argoproj.github.io/argo-cd/user-guide/status-badge/

https://github.com/argoproj-labs/argocd-notifications/releases/tag/v1.0.0

the "oncePer" feature looks like it could be useful for us

Currently we use inotifyd to watch for changes in /ircd/
We do want to fire on e.g. /ircd/ircd.yaml and /ircd/tls/tls.crt
We dont want to fire on e.g. /ircd/db/foo changes

At the moment I think its either not crossing device boundaries or not descending into sub directories.

We may need to move around our mount points to resolve this correctly.

v1beta1 is now dropped. I updated the ingress-nginx Kustomization. It looks like a maintenance script updated the live cluster already, but this needs to be done at some point to avoid overwriting the fixes.

I have a bit of a pet hate for kibana and to a less degree, elastic-search. Lets set up loki (to be used from grafana).

TODO list:

• collect logs (fluent-bit vs promtail?)
• rotate logs (? is this automatic?)
• make logs searchable
• collect node logs (e.g. dmesg)
• look into @fatalbanana 's ideas below (boltdb?)
• set up ingress for local logcli usage

Use https://github.com/hashbang/docker-postfix ?

@benharri where does our config currently come from?

this should be a token of some sort that makes it so only some things can post to /signup

tcp ingress to the container on port 22, http[s] ingress goes to a static html file

https://github.com/coreos/kube-prometheus includes prometheus-operator as well as grafana, alertmanager, kube-state-metrics and more.

Consider replacing our prometheus-operator application with it.

• Note that they do include a kustomization.yaml in the root of their repository.
• Make sure we use a compatible release

I just merged ergochat/ergo#1111, which adds support for authentication plugins via subprocess invocation. The goal in this issue is to start managing IRC authentication credentials in userdb, then get everyone using SASL. Here's a tentative plan:

1. hashbangctl should generate a strong, secret password on new account creation (@benharri was uncomfortable with this? I definitely want to hear any objections out)
2. Write a script to check this password against userdb
3. Turn off account registration in oragono, and instead configure this script (in the new accounts.auth-script config block), with autocreate enabled
4. New account creation should automatically initialize ~/.weechat/irc.conf to use SASL PLAIN with the autogenerated password. We could write a script like enable-irc-sasl that does this for existing users.

Will be able to at least validate yaml.
Not sure if we'll be able to do a client side dry-run without exposing secrets to the CI server?