[tech] Tech/Wheel Meeting 2021-01-10 14:00 - One hour reminder

Sun Jan 10 13:43:10 AWST 2021

Apologies, I won't be able to make it to today's meeting as I have other 
obligations.

Regards,

Coffee [CFE]

On 10/01/2021 1:00 pm, root wrote:
> Wheel Meeting Agenda - Sunday 2021-01-10 14:00
> ==============================================
> 	- VENUE: UCC Clubroom
> 	  - and online at https://meetings.ucc.asn.au/b/bob-yrk-uy6 ?
>
> *Meeting opened xx:xx*
>
> ## Attendance
> - Present
> - Apologies
> - Absent
>
> ## Next meeting
> - Schedule next meeting
>    - what's happening before O-Day 2021-02-19 ?
>    - ACTION: xxx, who hasn't tried it recently?:
>      - Update the agenda, update the crontab, check at T-7days that the notice really went out
>      - Set and verify reminders of next meeting: `motsugo# crontab -e`
>        - skip the `4day` , unless there's issues at `1week` ?
> - Curate agenda.next
>
> ## Standing items (brief)
>
> ### Visibly reinduct members new (and old?) with the "Wheel Group Ethical Guidelines"
>    - examining an Ethical Guideline, e.g. asking:
>      - What's an example situation in which it could be encountered?
>      - What other guidelines or rules could it conflict with? How would
>        one resolve it?
>
> ### Status check: Regular updates, monitoring
>    - atlantic cluster
>      - should we `apt upgrade` ? do some reboots?
>    - e.g. Debian oldstable 9 "stretch" -> Debian stable 10 "buster"
>      - find candidates on ocsinventory
>        - has it stopped reporting versions in Debian 10?
>          https://ocsinventory.ucc.asn.au/ocsreports/index.php?function=visu_search&fields=HARDWARE-LASTCOME&comp=tall&values=07/07/2020%2007:19&values2=&type_field=
>    - molmol
>      - Dead SSD? at Mon  9 Nov 08:00:10 AWST 2020
>        ```
>        molmol: /space/scratch/nick>zpool status|grep -C4 DEGRADED
>        logs
>          mirror-4                       DEGRADED     0     0     0
>            ada0p3                       ONLINE       0     0     0
>            3087349144323640050          UNAVAIL      0     0     0  was /dev/gpt/molmol-slog0
>        ```
>      - Upgrade and performance analysis:
>        1. monitoring: add prometheus metric export
>        2. iozone performance-and-latency-under-load benchmark
>        3. enable metaslab debugging mode: https://serverfault.com/questions/511154/zfs-performance-do-i-need-to-keep-free-space-in-a-pool-or-a-file-system
>        4. iozone performance-and-latency-under-load benchmark
>        5. OS upgrade
>        6. iozone performance-and-latency-under-load benchmark
>
> ### Status check: Backups
>
> ### Status check: Password/Key rotations
>    - https://en.wikipedia.org/wiki/Pro_re_nata
>
> ## ..._then_ New wheel members, additions, nominations
> - Welcome to wheel!
>    - Read /home/wheel/docs/WelcomeToWheel
> - winadmin, sprocket
>    - [BRD]@2020-08-13 `uid=12426(bird) gid=10021(gumby) groups=10021(gumby),10069(committee),12203(door),666(winadmin),777(sprocket)`
>    - `uid=12469(hilmi) gid=10021(gumby) groups=10021(gumby)`
>
> ## New Matters
> - [NTU] `Dec  8 05:41:46 motsugo kernel: [10920663.319201] Oops: 0000 [#1] SMP PTI`
>    - thanks for the remote power cycle, [TPG]
>    - `murasoi:/var/log/ucc/messages` is pretty noisy!
>      - can we fix some of the root causes of the noise?
>      - kerosene's timezone is UTC, probably should be local?
> - [TRS]@2020-11-03: SOGo ( https://sogo.nu/ ) has been down for a while too
>    - [NTU] molmol had stopped responding - out of memory and the wrong thing got killed?
>      - remote power cycle of molmol
>      - OOM possibly triggered by rdiff-backup on huge files?
>        - ACTION: [???] clean up the huge files, see `mollitz:/backups/log`
>      - is SOGo working again?
>        - ACTION: [???] can we add a prometheus+grafana health check for SOGo?
> - [NTU] Live demo: ceph magikarp:osd.3 is smaller than it should be, let's fix it
>
> ## Matters arising previously
>
> - ACTION: xxx, who hasn't tried it recently?: Set and verify reminders of next meeting: `motsugo# crontab -e`
> - ACTION: DONE? [MTL]+[MPT] poking zonemake.py and its API-driven replacements and children
>    - ACTION: DONE? [MPT] cf_tools / zonemake.py / octodns: generate API tokens for uccpass
>    - https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-December/005411.html
>    - [NTU] How can internal-only proxmox cluster VM hosts automate new letsencrypt certs?
> - ACTION: [MTL] to look at UCC web reverse proxies
> - ACTION: [MPT] UWA IT liason: matrix test domain
> - ACTION: [MPT] update https://wiki.ucc.asn.au/Network with latest traffic paths
> - ACTION: [TEC] to look at dashboards for murasoi network traffic
>    - [NTU] can it capture when bulk TCP resets are sent by upstream connection-tracking routers?
>      - graph the age of existing non-LAN connections, look for spikes to zero?
>      - graph new connections, look for spikes?
>
> *Meeting closed xx:xx*
>
> ----
>
> ```
> # https://demo.hedgedoc.org/Hlsapf47RsqpgIjqLVfMUw
> cd /home/wheel/docs/meetings
> CODIMD_SERVER=https://demo.hedgedoc.org codimd export --md Hlsapf47RsqpgIjqLVfMUw ./$(date +%Y-%m-%d).txt
> git commit -a "minutes"
> ```
>
> # vim: tabstop=4 shiftwidth=4 expandtab
> _______________________________________________
> List Archives: http://lists.ucc.asn.au/pipermail/tech
>
> Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/coffee%40ucc.asn.au

-- 
| Zack Wong           <coffee at ucc.gu.uwa.edu.au> |
| UCC Wheel Member                               |
--------------------------------------------------
[ A computer without COBOL and FORTRAN is like a ]
[ chocolate cake without ketchup and mustard.    ]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.ucc.gu.uwa.edu.au/pipermail/tech/attachments/20210110/5439bf39/attachment.htm>