[tech] Minutes of Wheel/Tech Meeting 2014-05-09
matches at ucc.asn.au
matches at ucc.asn.au
Fri May 9 20:20:21 WST 2014
Wheel/Tech Meeting 2014-05-09
Wheel: [SZM] [SLX] [BOB] [TPG] [MTL] [BG3] [GOZ] [NTU] [*OX]
Others: [GEE] [VIJ] Alice (Simplebard), [DOM] [JDN]
The Agenda for the meeting is
------------------------------
- A SSOE that can be used for things like Learn2Linux and Intro to
Programming
- Wether we want it, what sort of environment it would be etc etc.
- Rules for Colos and VMs and maybe some sort of network usage monitor
- The UPS, what happened, and it's current state
- What (important) machines need upgrading/replacing in the next year
- The NetApp that was donated to us by NetApp
- Neatening up the Machine room - Making cables easier to trace
- And of course, Building the new File Server
- Anything else people want to talk about [things added during meeting]
- Password Escrow for Wheel
- [HMC] wants to know about the Snack Machine firmware, does anyone
have a copy of the code?
- Committee wants to look into running a VM to provide other clubs
with mailing lists, preferably on their own domains
- Donation of Virtual Routers
Meeting opened at 18:11 by [BOB]
- [BOB] explains what meeting is
A SSOE that can be used for things like Learn2Linux and Intro to
Programming
----------------------------------------------------------------------------
- [SLX] explains it; it is a boring standard environment that isn't
Redhat
- To be used for events so all machines are running the same thing
- We currently just get everyone to ssh to motsugo (PuTTY for
Windows) to get an SOE
- Downside: No graphics
- Haven't had any graphics events yet but might (eg: Intro to
OpenGL)
- [BOB] who wants to be involved?
- [SLX] if it happens (not sure it should)
[*OX] arrives 18:15
- VirtualBox raised - people have had problems
- [SLX] 1. Need to login and start VM, takes time, 2. ...
- [TPG] Fix the thinterms?
- They are broken for <reasons>
- [BOB] whatever we do has to be quick and easy and Just Work
- [*OX] doesn't think so...
- A lot of events are targeted at having members participate in
the club
- It teaches people how to deal with the club
- [TPG] Netboot environment? Not the same as thinterms because they
are evil
- They are evil
- [BOB] next event?
- [GOZ]'s programming talks, but these have been cancelled until
next semester
- [BG3] When will we have the next meeting?
- [SLX] We can talk about after this meeting
- [BOB] Netbook, LiveUSB or LiveCD
Rules for Colos and VMs and maybe some sort of network usage monitor
--------------------------------------------------------------------
- We had an incident that coincidentally happened at the same time as
heartbleed but was unrelated
- At the same time we actually had a heartbleed issue though but we
will ignore that one
- People need good passwords
- People need to patch their machines
- [BOB] thinks they are similar problems, for most intents and
purposes they are the same thing
- [TPG] policy "Wheel must have a key so they can run password checks"
- [BOB] recommends Cronjob for Jack The Ripper
- Asks for volunteers
- How do commercial VM providers deal with it?
- They nullroute (drop all packets from VM)
- [SLX] Recommends a "After 24 hours if you haven't patched it and we
can tell, block all ports" policy
- Can things be done in Proxmox?
- Detection?
- Network monitoring
- [BOB] setup VMs are on different VLANs so we can pass different
ranges through filtering
- [ASH] ignored this (using IPs on clubroom range)
- Discussion of fixing it for [ASH]
- Too many VMs
- What do we need for Network Monitoring?
- [BOB] Netblocks Blue
- [TRS] says it is glorified IPTABLES (says [BOB], [TRS] not here)
- [BOB] says it works
- [SLX] an alert when a machine does lots of traffic would be good
- [TPG] how do you tell it's not just an iso
- Mitigation?
- [TPG] Anti source spoofing, do we do it, we should
- [*OX] suggests using the Packeteer to restrict things to 100Mb
inside
- Again, how do we tell it isn't legitimate traffic (eg: LaTeX
downloads)
- [TPG]
- Packeteer upstream of VMs
- Murasoi monitoring traffic
- Call for volunteers
- [VIJ] any hardware?
- Only the packeteer
- Call for volunteers? [BOB] says to look at Netblocks Blue
- It can detect other things (Brute force, Virus, Wierd Stuff)
- [TPG] Best solution is connection limiting on Murasoi.
- But people need to react, we need more automation
- Rules!
- Rules!
- Wheel members need access to a VM. Easy access. Not "technically
possible".
- A key. That has to be on VMs.
- [*OX] how would this work for Windows? A written down password?
- People want to ignore windows
- Linux VMs need to have a key on it, wheel will check it is there
periodically
- [BOB] Improve documentation
- Explain things to people with Colo machines
- [SZM] had tried to do this in the case that brought all this up
- Jack The Ripper is better than an external attack because it
doesn't get slowed by Fail2Ban
- People want to vote on a policy that Wheel Members need root
access to a machine
- Discussion over whether it should be all of wheel or just 1, 2,
etc...
- People might not trust all of wheel
- People might pick wheel members that are no longer active
- [TPG] Wheel members can get in if they want to anyway, no point
restricting to just one wheel member
- [NTU] access needs to be logged and the owner informed
- [BOB] alternative to wheel keys is giving a root password to wheel
- That makes automated checks harder
- Standard ISOs! Standard OS! Puppet is mentioned...
- Uh oh
- Abort!
- [SLX] motion: For a machine that runs SSH, that the user does not
object, either all wheel keys or a specific key which all wheel members
have access to ...
- Discussion over keys vs key
- [*OX] VM Manager user...
- CIRCLES DETECTED
- CIRCULAR MOTION IS MADE
- Or not
- Wheel members can get around the circles
- Logs are discussed. And found to be imaterial.
- [BOB] amend motion so that Wheel members don't do anything unless
it is an emergency
- [SLX] Wheel members need access to machines, if this is too hard
or people complain we will discuss on a case by case basis.
- All in favour
[DAA] arrives 18:44
- [BOB] wants to make another rule...
- Only members can have shell access on UCC machines or member VMs
- [DAA] Controversial!
- Controversy ensues
- Some people are giving shell access to VMs to friends or group
members who don't really need it and aren't members
- [DAA] can Fish have an exception?
- It does make Group Projects annoying
- [TPG] suggests requiring emails to Wheel
- [SLX] wants to talk about Fish
- If there are any non members accessed, wheel needs to know
- [*OX] wants to find out who is doing this. [BOB] points finger
at [LAW].
- Retroactively Unanimous (the wheel needs to know, not the
pointing fingers at [LAW])
- [SZM] can wheel make policies? How do we convince people they are
actually rules? They'll probably ignore us
- [DAA] says to send them to committee to endorse
- [SZM] agrees
- [MTL] and [*OX] say we can just turn the VMs off
- [BOB] says committee is scared of making these sorts of policies
The UPS, what happened, and it's current state
----------------------------------------------
- [BG3] It charges to about 50% but dies and drains
- [BOB] thinks batteries are shorted, [BG3] says they are fine
- It is losing mains power basically
- [BOB] thinks it is batteries, will pull apart and check
- Built the battery packs in 2010
- They are probably screwed
- Power point is fine
- Do we want another one?
- It has been useful a couple of times
- People talk about why UPSs are good
- Servers not instantly turning off is good in many ways
- RAM the caches
- Don't need as much beef
- Would be better off making machines cooler
- Paint them red
- There are 36 batteries in 3 packs. Or maybe 8 in a pack. So 24.
- $480 from Altronics
- [*OX] when was the last time we didn't lose power due to UWA
cutting the power intentionally?
- Eg: UWA cut power whilst we were at the camp
- We have two circuits, we should use both
- We do (usually) get advance notice. Sometimes no one prepares.
- 30 seconds counts...
- Caching!
- Smaller UPS for individual machine (fileserver)
- [BOB] if batteries are the only problem, spend the $400
- Will suggest to committee
- [BOB] and [BG3] to deal with
Interlude discussing how long the minutes are
[MTL] leaves at 18:57
Back to UPS
- Is there monitoring software?
- The software is for Windows XP, have fun
- We should un rack mount it
- [BOB] hints at people not on wheel fixing it
- Yay!
- [VIJ] foolishly mentions interest
What machines need upgrading/replacing in the next year
-------------------------------------------------------
- The Fileserver!
- It isn't here yet! (missing cable for MiniSAS backplane -> main
board)
- Parts will be here in a week
- What to put on it? FreeNAS, ZFS, various scary things are
suggested
- [SLX] We tried ZFS on Red, it worked until everyone fucked it up
- [DAA] ZFS is curiously anticlimactic (it just works??)
- [BOB] thinks not
- [SLX] (does not) recommend BTRFS
- [DAA] we have good backups now. They actually work. 150Mbs
overnight
- Everything except vmstores
- [*OX] other options,
- FreeNAS
- OpenFILE (linux based, [DAA] says its shit don't use it)
- Solaris based (OpenIndianda) - - No
- Discussion of "Pirate Debian", breaks the GPL or something?
- Just a linux server with LVM?
- Boring but functional
- ([SZM] would settle for boring but functional in a
fileserver...)
- [SLX] The people that shout the most about ZFS can configure it
- Does anyone have any other suggestions?
- Windows (hahaha)
- FreeNAS
- LibreNAS (???)
- FreeNAS FreeNAS FreeNAS
- Web interface (!? Shitttt)
- [DAA] it is FreeBSD with something nice that is awful that is
not terribly bad
- [TPG] can always kill with fire
- If it can do the things we want...
- [TPG] Fileserver seperate from domain master (SAMBA)
- We want to get things off Mylah
- People want to kill Mylah
- [BOB] wants to kill Mylah, then put Mylah on Mylah as a VM
- Even though we have old machines like Mylah they are pretty
good.
- Only machine running out of RAM is Medico with all its VMs
- Discussion of RAM usage efficiency due to VM distros
- [DAA] does Mylah support hypervirtualisation?
- We aren't sure. [BOB] checks right now.
- [SLX] we will get rid of Mylah as a logical machine and what it
was physically on may become a VM host
- Trial the new proxmox maybe
- [DAA] we will need to put the SAMBA domain master on something
else
- [BOB] likes Mylah as a host, just virtualise it
- Mylah was virtualised at one point in one of its 9999 lives...
- It got unvirtualised because of reasons (possibly because of
running VMs on VMs?)
- What are we putting on the fileserver?
- [BOB] Someone does tests and emails wheel and then we argue
and then do what they suggested anyway
- [*OX] we install FreeNAS tonight and are done
- When [BOB] picked Proxmox he tried other things first
- [*OX] we try FreeNAS tonight and then we move on
- Back to Mylah
- We want to kill it to reduce peak heat loads
- What do we replace it with
- Somehow we are at boosting medico to run more VMs on it
- Hang on, don't VMs still make heat
- Concensus is that Medico will be less hot even running
things as VMs than the VMs would be as not VMs
- Camwhore is being virtualised TONIGHT!
- [BOB] asks how?
- By "Virtualising" [BG3] may not mean making a VM, just moving
to something else (?)
- Murasoi is the next machine to have its right to exist debated
- It was dying because its temperature cutout was low
- It is also in the hottest spot
- [BOB] stopped it dying by increasing the cutout until it
stopped dying
- When it is on fire we will blame [BOB]
- Murasoi is deemed to be worthy of existing as a non-virtual
machine (NVM)
- Distractions happen
- Mooneye's fate?
- Who is brave enough to attack it
- It runs mail, bind, the wiki
- [BOB] threatens to replace it with a raspberry pi
- Mooneye IS the webserver (ucc.asn.au) contrary to popular
belief
- SAMBA
- Back on Mylah
- [BOB] moves to buy another Medico or similar and that when/if
we get it, we get rid of Mylah and put Mylah on the new VM host as a VM
- It will be cool, redundant, capacious, kills many birds with
one stone, much server
- [DAA] just make the new VM host mylah, don't bother VMifying
- Mylah does more things than people realise, this could
cause problems if it is a VM (it NFS forwards the SAN)
- Motion amended by [SZM] to not specify what will happen to
Mylah
- Silently. Without shouting.
- [SLX] Motion is: We will chuck out Mylah as soon as we can.
- Unanimous
- Motion to get a second VM host: Unonimous
- [DAA] Discussion about Manbo in 2004 having 16G of RAM and
16CPUs
- Historical information! We had to take some out because it
was too big, the signal took too long to get from one side to the other
Sideways tracking occurs
- Can we get a 10Gb card for Bitumen
- SAN is fast (2Gb)
- Card for Bitumen is $$$$
- Some guy that [GOZ] can't remember the name of...
- Ben? Brad!
- ... he still has a bunch that they throw out all the time
- [BOB] will take 3
- SAN is slow (100Mb) because of PCI Bus? Or Mitch-tech
- Something something
- And then someone suggested we get Food! Not :(
- [VIJ] volunteers to make a network map.
- Black isn't black
- Slightly darker black is the new black
- [BOB] We should get a black permanent marker that is actually
black
[SLX] clubroom machines are breaking
- [BOB] says to not install Debian...
- Religious war ensues!
- Religious war continues!
- [SLX] Lets not argue about the distros...
- Scientific Linux is the ONE TRUE DISTRO
- Distro war continues
- Shouting happens
- We agree that AMD's proprietry driver is crap and we will not
install it
The NetApp that was donated to us by NetApp
-------------------------------------------
- FAS2020
- [BOB] says to use it as /scratch
- [TPG] keep half disks as cold spares
- [BOB] could replace disks on other NetApp (some died, power surge?)
- We have all the licenses
- [*OX] is going to setup a mirror
- Discussion of Netapp breeding program
- It has been a LONG meeting
- We don't really need more space.
- [SZM] let's let someone who is interested do what they want as a
NetApp, and move on
- People agree
- Other people want to use the NetApp for backups
- [TRS] How will you feel about a FAS2020 in the server room?
- [TPG] wants to backup the coke database infinitely, INFINITELY
- Discussion of what time interval is sufficient
- Discussion of why we want the coke database to go back
infinitely or possibly less
Donation of Virtual Routers
---------------------------
- Yes
"Back in my day" stories ensue
- [SLX] We had to carry the parity bits up the hill both ways
- [DAA] "You wouldn't download a router"
- [BG3] talked to people at Whackhon
- Yes
- Yes
- Yes
- yes | yes | less
- The nods have it
[*OX]'s things
- Fix OCSInventory
- Cry about mantis
- Bitch about Sprocket / Winadmin / Coke / Door
- Uh oh... policies detected
- Committee will be merging coke and door maybe. There is now a
LOLCATDOG.
- Discussion of whether wheel members are by definition on door
- Committee members are in favour, non committee members are not -
wheel isn't entirely active members and aren't by definition valuable
door members
- [SLX] TO THE LISTS
- Wheal don't care
Password Escrow for Wheel
-------------------------
- Escrow not Escroe
- Crow is my favourite character from a video game
- [DAA] both times VMs got rooted in living memory was shitty
passwords. There are 14 people with passwords that broke in under a
minute.
- Distraction whilst people look at list of crappy passwords
- [GOZ]'s brother is shamed
- [NTU] and [*OX] discuss
- [NTU] can we log admin access to VMs?
- [BOB] Proxmox web interface logs things.
- This won't work for ssh keys, the things we wanted to put on
VMs so wheel had access
- [*OX] "You wouldn't download a car Bob"
- I don't get it...
- [BOB] Hang on, what actually is an s-crow?
- [NTU] "I'm keeping the password in a safe place in case you can't
get to me"
- Eg: Escrow software
- Eg: Money transfers
- Should we have a central password vault (slightly different
concept?)
- [*OX] was going to talk but gave up, he's gone
- He's back
- Other people talk
- [NTU] File store on dedicated raspberry pi or something
- [DAA] put it on Mooneye, if Mooneye is fucked we're all fucked
anyway
- Bikeshedding occurs
- raspberry pi vs mooneye; raspberry pi = minimal power
- ... It's not like we'd shut off Mooneye (is it? Refer back to
fate of mooneye topic?)
[*OX] leaves at 19:48
- On the note of "Minimal Power" we should have a red notebook and
write all the passwords in it and padlock it in a box in the machine
room
Discussion of GPG and PGP
- GNU and not GNU
[DAA] leaves at 19:50 because he has a life
- Mylah does not support hypervirtualisation
- [BOB] checked already
Central password store
- [BOB] talks about hierarchy of trust... you get passwords as you
need them on Wheel not straight away
- Discussion of changing passwords
[BG3] Next Meeting
- 3rd week of second semester
[BOB] Who is doing Camp Network?
- [BOB] and [TPG] will be away
- [SLX], [GOZ] volunteer
- [BOB] lectures about preparing with meetings
[LAW] arrives at 19:57
- [TPG] Remember Beatentrack == Cabellera with different hard disk
- Discussion of caching steam apps
- Deffered to preparation meeting
- Or not
That's all there is.
There isn't anymore.
Except GitHub.
Meeting Closed at 20:00
- [BOB] to organise pizza run
- [BOB] says "Fuck No"
- Someone who cares to organise pizza run
- [JDN] volunteers! And is welcomed to wheel! (not really)
Also available at:
http://www.ucc.asn.au/infobase/minutes/2014/2014-05-09.tech
More information about the tech
mailing list