[tech] Tech/Wheel Meeting 2020-12-06 14:00 - 24 hour reminder

Sun Dec 6 08:53:38 AWST 2020

Much to discuss? I’m training 12-2 but could come after?

> On 6 Dec 2020, at 8:50 am, Susie Johnston <susie at ucc.asn.au> wrote:
> 
> Much to discuss? I’m training 12-2 but could come after?
> 
>> On 5 Dec 2020, at 4:57 pm, root <root at ucc.gu.uwa.edu.au> wrote:
>> 
>> Wheel Meeting Agenda - Sunday 2020-12-06 14:00
>> ==============================================
>>   - VENUE: UCC Clubroom
>>     - and online at https://meetings.ucc.asn.au/b/bob-yrk-uy6 ?
>> 
>> *Meeting opened xx:xx*
>> 
>> ## Attendance
>> - Present
>> - Apologies
>> - [NTU] - may be online
>> - [THA]
>> - Absent
>> 
>> ## Next meeting
>> - Schedule next meeting
>> - what's happening before O-Day 2021-02-19 ?
>> - ACTION: xxx, who hasn't tried it recently?:
>>   - Update the agenda, update the crontab, check at T-7days that the notice really went out
>>   - Set and verify reminders of next meeting: `motsugo# crontab -e`
>>     - skip the `4day` , unless there's issues at `1week` ?
>> - Curate agenda.next
>> 
>> ## Standing items (brief)
>> 
>> ### Visibly reinduct members new (and old?) with the "Wheel Group Ethical Guidelines"
>> - examining an Ethical Guideline, e.g. asking:
>>   - What's an example situation in which it could be encountered?
>>   - What other guidelines or rules could it conflict with? How would
>>     one resolve it?
>> 
>> ### Status check: Regular updates, monitoring
>> - e.g. Debian oldstable 9 "stretch" -> Debian stable 10 "buster"
>>   - find candidates on ocsinventory
>>     - has it stopped reporting versions in Debian 10?
>>       https://ocsinventory.ucc.asn.au/ocsreports/index.php?function=visu_search&fields=HARDWARE-LASTCOME&comp=tall&values=07/07/2020%2007:19&values2=&type_field=
>> - molmol
>>   - Dead SSD? at Mon  9 Nov 08:00:10 AWST 2020
>>     ```
>>     molmol: /space/scratch/nick>zpool status|grep -C4 DEGRADED
>>     logs
>>       mirror-4                       DEGRADED     0     0     0
>>         ada0p3                       ONLINE       0     0     0
>>         3087349144323640050          UNAVAIL      0     0     0  was /dev/gpt/molmol-slog0
>>     ```
>>   - Upgrade and performance analysis:
>>     1. monitoring: add prometheus metric export
>>     2. iozone performance-and-latency-under-load benchmark
>>     3. enable metaslab debugging mode: https://serverfault.com/questions/511154/zfs-performance-do-i-need-to-keep-free-space-in-a-pool-or-a-file-system
>>     4. iozone performance-and-latency-under-load benchmark
>>     5. OS upgrade
>>     6. iozone performance-and-latency-under-load benchmark
>> 
>> ### Status check: Backups
>> - https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-December/005410.html
>> - Legacy backups: mollitz
>>     - Prometheus metrics for uccmonitor (assistance welcome)
>>       - hopefully just a https://gitlab.ucc.asn.au/ucc-systems/ansiblemonitoring away
>>     - a proper packaged install of its old tools like megaclisas-status (assistance welcome)
>> - New backups
>>   - ACTION: [NTU] order drives
>> 
>> ### Status check: Password/Key rotations
>> - https://en.wikipedia.org/wiki/Pro_re_nata
>> - time for a `john(8)` run
>> 
>> ## ..._then_ New wheel members, additions, nominations
>> - Welcome to wheel!
>> - Read /home/wheel/docs/WelcomeToWheel
>> - winadmin, sprocket
>> - [BRD]@2020-08-13 `uid=12426(bird) gid=10021(gumby) groups=10021(gumby),10069(committee),12203(door),666(winadmin),777(sprocket)`
>> - `uid=12469(hilmi) gid=10021(gumby) groups=10021(gumby)`
>> 
>> ## New Matters
>> - [TRS]@2020-11-03: SOGo ( https://sogo.nu/ ) has been down for a while too
>> - [NTU] molmol had stopped responding - out of memory and the wrong thing got killed?
>>   - remote power cycle of molmol
>>   - OOM possibly triggered by rdiff-backup on huge files?
>>     - ACTION: [???] clean up the huge files, see `mollitz:/backups/log`
>>   - is SOGo working again?
>>     - ACTION: [???] can we add a prometheus+grafana health check for SOGo?
>> 
>> ## Matters arising previously
>> 
>> - ACTION: xxx, who hasn't tried it recently?: Set and verify reminders of next meeting: `motsugo# crontab -e`
>> - ACTION: DONE? [MTL]+[MPT] poking zonemake.py and its API-driven replacements and children
>> - ACTION: DONE? [MPT] cf_tools / zonemake.py / octodns: generate API tokens for uccpass
>> - https://lists.ucc.gu.uwa.edu.au/pipermail/tech/2020-December/005411.html
>> - [NTU] How can internal-only proxmox cluster VM hosts automate new letsencrypt certs?
>> - ACTION: [MTL] to look at UCC web reverse proxies
>> - ACTION: [MPT] UWA IT liason: matrix test domain
>> - ACTION: [MPT] update https://wiki.ucc.asn.au/Network with latest traffic paths
>> - ACTION: [TEC] to look at dashboards for murasoi network traffic
>> - [NTU] can it capture when bulk TCP resets are sent by upstream connection-tracking routers?
>>   - graph the age of existing non-LAN connections, look for spikes to zero?
>>   - graph new connections, look for spikes?
>> 
>> *Meeting closed xx:xx*
>> 
>> ----
>> 
>> ```
>> # https://demo.codimd.org/Hlsapf47RsqpgIjqLVfMUw
>> cd /home/wheel/docs/meetings
>> CODIMD_SERVER=https://demo.codimd.org codimd export --md Hlsapf47RsqpgIjqLVfMUw ./$(date +%Y-%m-%d).txt
>> git commit -a "minutes"
>> ```
>> 
>> # vim: tabstop=4 shiftwidth=4 expandtab
>> _______________________________________________
>> List Archives: http://lists.ucc.asn.au/pipermail/tech
>> 
>> Unsubscribe here: https://lists.ucc.gu.uwa.edu.au/mailman/options/tech/susie%40ucc.gu.uwa.edu.au