[tech] Tech/Wheel Meeting 2025-06-16 18:30 - One hour reminder

[tech-reminder at ucc.asn.au]

Tech/Wheel Meeting Agenda - Monday 2025-06-16T18:30
===================================================
- VENUE: UCC Clubroom
  - and online at https://meetings.ucc.asn.au/b/tech

*Meeting opened HH:MM*

## Attendance
- Present
- Apologies
- Absent

## Next meeting
- Schedule next meeting
  - *day 2025-mm-ddTHH:MM
- ACTION: [???] shall be this meeting's secretary! This entails recording minutes for meeting n (beware mid-meeting glitches) and ensuring meeting n+1 reminders succeed:
  - Checklist follows:
    - Clone a new issue from [[https://gitlab.ucc.asn.au/UCC/tech-todo-list/-/issues/32]]
      - Preferably immediately; then reopen it and assign it to yourself
      - This issue is to keep track of any async secretarial duties detailed ahead
      - Type `/clone` into the "Write a comment" box as a "quick action"
      - Update the title to match today's date
  - [ ] ACTION: Save and commit the minutes of today's meeting, during the meeting; and at the end
  - [ ] ACTION: Set and (later) verify reminders of next meeting:
    - [ ] Promptly update `agenda.next` with the TIME/DATE/VENUE
    - [ ] Perform initial curation of `agenda.next`, and move any previous or longstanding action items out of it and into GitLab (see Action Items section below)
    - [ ] Update the crontab: `motsugo# crontab -e`
    - [ ] Check at T-7days that the notice really went out, fix for T-4days if needed
- [ ] Everyone, before next meeting: Curate `agenda.next`, and move any items you think should be tracked as GitLab issues into GitLab issues, as above

## Optional items - choose at the start of the meeting
- [ ] Ethical guidelines
- [ ] Monitoring
- [ ] Backups
- [ ] Password rotations
- [ ] New members
- [ ] Quick check of ChangeLog
- [ ] Lessons learnt

## Current Action Items
### Boilerplate/Review
- Review GitLab bug list: [[https://gitlab.ucc.asn.au/UCC/tech-todo-list/-/issues/]]
- Briefly discuss anything iff needed, but don't spend too long rehashing previous meetings ;)
- Then:
  - New actions/ACTION items: put them in the minutes once, but add to bug list to minimise repeat discussion
  - Ongoing actions: don't keep them in the agenda, summarise updates - iff needed
  - Completed actions: mention in the agenda that it's been completed, summarise - iff needed

### Action items to discuss
- ACTION: All clean up the below action items and move longstanding ones into GitLab issues above.
- ACTION: [GPO] [I2N]: Pester UWAIT about firewalled IPs
- ACTION: [MTL] [MPT] [GPO] [NTU]: Update ucc-mailman to mailman3
    - Tangential - "user-webserver" Ansible role is nearly ready, will help with long term replacemnt of Debian 10 systems and decoupling stuff from here.
    - Debian upgrades also required for this.
    - [ROY] Watch out for "/bin" "/usr/bin" [merge](https://wiki.debian.org/UsrMerge).
- ACTION: [MTL] [MPT] [GPO]: Seperate mussel's functions into seperate functions, separate VMs
    - See notes above for ucc-mailman.
    - Ansiblising everything.
- ACTION: [GPO] [NTU] [TWA] to rotate uccpass passwords
    - Nope. Nothing.
    - [NTU] Let's pick a time.
    - [NTU] Seeks volunteers.
    - [TWA] Volunteers.
- ACTION: [GPO] [MPT] to fix OpenVPN and look at other VPN setups
    - Had a look at it again, recently, but no idea why it's broken so far.
    - Jasper: suggests disabling IPv6 (probably not the issue).
- ACTION: [333] to fix the speed of uccroot `push.sh`
    - Done 2025-05-05 :) 
- ACTION: ALL to add changes or work you have done to https://wiki.ucc.asn.au/ChangeLog
- ACTION: [333] (to draft and send) [BIRD] (to proofread) to send an email to @wheel for reminding proper good'ol days documentation & communication
    - No progress, sorry.
- ACTION: [BIRD] [333] to update welcome-to-wheel document.
    - No progress, sorry.
- ACTION: [TPG] [I2N] [ROY] [MTL] [ZPH]: Investigate further integration of SMART Monitoring, network monitoring, and the general fixing up `grafana`.
    - `molmol` ZFS health
    - `wobbegong` Proxmox Backup
    - Monitoring `ceph` SMART Status
    - Temperatures!
    - [ZPH] No progress yet. Looking into ElasticSearch, and getting it to pull data from Grafana and Prometheus.
        - [GPO] Why not not pull data straight into ElasticSearch, and have the visualisations use that as the backend instead?
        - Jasper suggests having Splunk (instead of syslog) forwarding logs into ElasticSearch.
        - ELK stack = Elastic search, Logstash, and Kibana.
        - [NTU] Adding ELK will not replace Grafana, so someone still needs to give Grafana some TLC (tender love and care, not three layer cell).
        - [ZPH] Currently playing with it as a PoC in a sandbox environemnt.
    - [ZPH] https://suricata.io/
        - Passively sniffs packets and compares them to a threat signature database, and sends alerts when threats are flagged.
        - [333] Sounds similar to Snode and Claroty that we use at work.
        - [TWA] Could look at offloading packet duplication to the NICs instead of CPU doing it.
- ACTION [NTU] [ROY]: Test our backup resiliency. Sort out date for workshop and tell rest of wheel.
    - Not done.
    - [NTU] seeking to schedule a time. 
    - [ROY] suggests after exams.
- ACTION: ??? to organise storage replacements and capacity exansions for Ceph cluster.
    - It is slowing down quite a lot, because we have a drive failing in Maltair
    - Flagging spurious read errors.
    - We should fix - *very soon*.
    - OSD.2, btw.
    - [GPO] We're starting to genuinely run out of space on vmstore.
- ACTION: ??? to fix CI/CD pipeline for DHCP and DNS is not working.
    - [GPO] Ran out of space recently, probably needs a bunch of old docker images cleaned out.
- ACTION: ??? to work on consolidating /home/wheel/docs into Wiki.
    - [ZPH] Lots of stuff we have in `/home/wheel/docs` is very outdated - like 2~3years outdated.
    - [GPO] How much really needs to be wheel docs, vs. being moved into the Wiki?
      - Very little
      - [NTU] There are a handful of files (Network, Backups, ...) that are kept on the wiki, and we can save a handy emergency snapshot to `/home/wheel/docs/wiki-snapshot`
    - [NTU] Also approves of moving more stuff to being only on the Wiki.
- ACTION: ??? to implement mitigations to prevent UCC website getting getting hammered:
    - Look at getting Cloudflare to tarpit AI scraper bots.
    - Enabled for secondary domains, but not `.asn.au$` nor `.gu(ild)?.uwa.edu.au$`
- ACTION: [333] to look into ACLs (if we're happy with them) to ensure appropriate permissions get applied/inherited in /home/wheel/docs/meetings, etc., even if the correct umask isn't set.

## Known Broken Stuff

## Matters arising previously

## Extra items (rename/refile as appropriate)
- [333] Donated a 1U Dell OEMR XL R640 server.
  - Shall we name it `mako`?
  - Thoughts on it being a PVE host, possibly replacing one of the old ones?
  - May also have a rail kit for it, but TBA.

*Meeting closed HH:MM*

----

```
cd /home/wheel/docs/meetings
wget -O ./$(date +%Y-%m-%d).txt https://hedgedoc.ucc.asn.au/t6u3DLnHTNyrnxR-AuYLGg/download
git add ./$(date +%Y-%m-%d).txt
git commit -m "Tech meeting minutes $(date +%Y-%m-%d)"
```

<!-- vim: tabstop=2 softtabstop=2 shiftwidth=2 expandtab
-->
<!-- Local Variables: -->
<!-- tab-width: 2 -->
<!-- End: -->