[tech] Tech meeting minutes

[Andrew Adamson]

A big thanks to [SZM] for taking the minutes below. They are also 
available to wheel members in wheel docs.

Further to these minutes, mussel was moved to a kvm VM on medico 
immediately after the meeting.

=======================================================================================

Attendance: [MRD] [SZM] [GOZ] [BOB] [DAA] [NTU] [DTK] [SLX] [MTL] [*OX] [TPG] [BG3] [HMC]

New Members: [BG3], [SAS] (not here)

[DAA] waves hands. Something about Xcode on napoli
[GOZ] wants to note that someone wants us to not assassinate people.
[SLX] shall we talk about SANs.
  - Enron and Stearns
  - Bad joke
  - [BOB] sent email about SAN (ex-ITS Backend Porn-(Administration) server)
  - Meeting derailed (again)
  - [DAA] recommends dsniff
  - Shelves 300G disks, 1 shelf OS, 176G disks
    - ~3TB in raid 10
  - Disks = heat
  - It is amazing
  - It is a block storage device not a file; need mylah - bottleneck
  - [TPG] recommends killing mylah
  - [BOB] beside the point; ports on controller have died, probably can't fix
    - Proprietary; so fuck them
  - Recommend Anti-Sponsors page for people that won't help us at all - we're looking at you, EMC
  - SFPs are broken, ports on enron are broken, "light spike" broke stuff, discussion of why they broke
  - Discussion of what SFPs we have
  - SAN does /away /mp3s and one of the /vmstores, but this doesn't matter "because we can get it off" <- [DAA] actually said this

[BOB] Netapp
  - Also proprietary
  - [MRD] they might still help
  - We don't have any contacts with them
  - [BOB] to email Mike Nye to try and get a contact
  - Some disks have died; not a major issue (yet)
  - They are special proprietary stuff
  - [DTK] says something about the Netapp being sensible
  - General condescending remarks
  - [NTU] we can grow and shrink the volumes if we run out of disks
  - We can lose 2 disks before we go "Oh shit" and 3 before it is fith
  - Something to look at (?)

[BOB] wants to make our own storage solutions that aren't proprietary
[SLX] put everything on Red
[BOB] want clustered storage (NFS or iSCSI or ???)
  - Discussion about what iSCSI is (a block device over IP, which lots of things can do)
  - [*OX] possibility of reusing SAN disks?
    - [BOB] they suck

[BOB] the machine room is hot; it's winter!
  - In summer, things will die
  - Turn off the colocated boxes
  - Bad
  - [NTU] reason we built the machine room; to cool the servers better
    - 5.1KW aircon = 5KW machines (in theory)
   - Discussion of chip box related cooling solutions
  - [NTU] we need to be able to shut stuff down if there is a temperature spike
    - [BOB] we shouldn't need to. Aircon has deice protection
    - [DAA] say the aircon catches on fire
  - [BOB] we need to reduce thermal load of machine room
    - Ditch the shitty gear
  - [MRD] send servers to equadorial embassy (leaves 18:52 - he is hungry)
  - [BOB] wants to kill the colocated machines :(
  - [*OX] we lost machines?
    - People: No
    - Other people: Yes
    - TODO: Count of machines
  - [SLX] Raspberry Pi is generating too much heat
  - [NTU] the SAN is probably to blame
  - [TPG] stick Raid 5 array in Pervirt (TODO: Rename that, please god)
    - [BOB] its name is mango
    - [BOB] it is a throw away box, don't use it for storage
    - Discussion of how shitty mango is
    - [SZM] why turn on mango if it is hot?
    - [BOB] it is very hot, but very fast, so turn it on
    - [TPG] put VMs on mango, kill it when it gets hot
  - [NTU] estimates 5min before machines die without aircon
    - [BOB] specifies it must be summer
  - People generally agree that stuff goes badly when things overheat
  - [BOB] can decide software implementation later; but right now... we want some sort of tiered storage
    - [*OX] Can we do cool things like feed it multiple ethernet cables
    - Yes
  - [BOB] we get a 3ru case with room for disks, 3 1/2, 2.5 dual/triple power supplies, motherboard, lots'o'RAM, battery backed RAM
    - Discussion
    - [*OX] it's not Industry
    - [MRD] industry has money
      - [BOB] it's proprietary
  - [DAA] Idea is: Get rid of SAN +/- NAS +/- motsugo
  - [TPG] need to work out airflow to machine room, dig out [JCF]'s thesis
    - Discussion of where things should go for the best airflow
  - [MRD] what if bitumen is the issue
  - [DAA] won't have a decision tonight
  - [TPG] priority is migrate stuff
  - [BOB] do people agree with me?
  - [DAA] doesn't care about heat in machine room. Thinks it's nice to have a diversity of things (netapp SAN) but accessing them is irritating.
    - Unify access to storage
    - Ceph - Clustering storage system
    - Phalic references
    - Bad joke
    - Would allow us to fully utilise things like NAS and SAN by treating as block devices
    - Disadvantages: Yet another layer
    - We should have 2 file servers
  - Talk about money
    - Will need to consult committee to decide if it is the best use of money, estimate $4.5K for custom server
  - [HMC] arrives 19:04
  - [BOB] we will learn more with a custom server
  - [DAA] price it up and get some comment on it
  - [BOB] distribute with that and the Netapp/SAN
    - Kill the SAN with fire
    - People hate the SAN
    - It is likened to a pile of crap
  - [BOB] Can we do multipath with the new server
    - [DAA] we can do it with a spanning tree, but we don't, because
  - [BOB] what happens if we don't use Ceph?
    - [DAA] drbd is the other thing. Works well with proxmox
    - Blobs on Filesystem on LVM on Raid Array on Block Device argument
      - Performance!
    - iSCSI can be done in proxmox
  - Should make it so that creating a VM has one interface
    - Proxmox is good for storage
    - [BOB] do we need something to manage fencing; high availability server in centre of cluster?
    - Something would be good for OS upgrades
  - [*OX] can we get rid of mylah

- Conscensus: We have finished talking about storage.
- [BOB] wants to look at Ceph
- Discussion of network limitations
  - Eventually we will have 10G
  - Eventually we will build UCC Tower
  - Some stuff [SZM] missed because power is low on [BG3]'s laptop 

- [SLX] mussel
  - Should we replace it?
  - What does it do? Everything? Web, radius, ldap (primary?) secure
  - [DAA] 2 types of complaints
    - 1. Too much stuff
    - 2. Too much cruft
  - [SLX] do we want all this core infrastructure on mussel to be on it (Is it still a user machine?)
    - [DAA] web needs to be on public machine
  - [DTK] A VM per service?
    - Most people disagree
    - Have a few groups
  - 19:15 - [GOZ] notes that Westminsterbongs didn't work
  - Argument about problems
  - Problems, problems, problems
  - Logic, logic, logic
  - Minutes, minutes, minutes
  - Hungry, Hungry, Hungry
  - Dreams about Unix Partitioning
  - The point [DAA] was making 6 minutes ago was that the problem is that when mussel crashes it shits people off. And it crashes because it has too much crap on it.
    - The OTHER problem is that at the moment it just seems to stop working sometimes
    

  - Move web and web related stuff off mussel
  - mantis is a VM that stuff might get moved to. Or maybe not.
  - [SLX] we also don't like mylah
    - We got it out of a public loo
    - [BOB] it is good tech (???)
    - [SLX] has nightmares about bulging batteries
    - Move SAMBA and LDAP to another machine
      - Not the same machine???
    - ABSLDJSAHDFIUWERIUWERKUASHDI7y
  - Pizza order
  - Funky mylah stopping the network?
  - [BOB] let's migrate mussel to KVM
    - Agreement!
  - [DAA] the 3rd problem is we have 3 differnt VM servers
    - We can't migrate motsugo KVM to proxmox

  - PIZZA Time 
  - Or not
  - Or yes
  - [DAA] this will take 5 minutes, I promise
    - General laughter

SAMBA 4
  - Migrate to samba 4 !
  - As you are all aware (?) SAMBA3 is the open implementation of windows 1997 stuff
  - NT3.0
  - Registers, registers, registry changes
  - Testament to microsoft's commitment to lol enterprise environments
  - People still use NT3, we pity them
  - NT3.1 had the start menu, one of them didn't
  - So...
  - SAMBA4 implements active directory. LDAP + Kerberos + Something else
    - Will make windows stuff much easier*
    - Deployment, group policy, make windows experience suck less
  - Problem: We have to throw away OpenLDAP
  - At the moment we have LDAP with SAMBA3 magic on top of it
    - For a long time we had 2 different systems
  - Problem: We would have to make major changes to config of all non windows machines
    - Just run magic tool on Windows machines
  - Linux stuff may work
  - SAMBA4 doesn't buy us anything we haven't got already
    - It may be a step backwards
  - We will move into the guild next year
  - We should redo the machine room by the way
  - Watch this space
  - [DAA] will show you terrifying stuff if you ask
    - Involves LDAP (easy) and Kerberos (net start)
    - Is it really Kerberos if it's not like using Kerberos?
    - Free Kerberos! (Yay?)
    - Is Kerberos the solution to our problems?
      - Maybe?
  - SAMBA 4 rewrite authentication system
  - Various people have suffered to bring us the current authentication system through a series of painful iterations
  - Stories about how LDAP used to work
  - I think it's been 15 minutes now
  - Web interfaces for things
  - How does this work with dispense? Maybe? Yes. Active directory
  - [*OX] just use dispense for authentication
  - [DAA] Ah, we can use the fish management system
    - [MTL] no that was some horror text based console game
    - [DAA] sounds about right

- Should committee pay for pizza?
  - Consensus: No

- Pizza order happening. [BOB] to coordinate. [BOB] says fuck no. Someone who cares to coordinate.

- I think I can stop now?
- People go start doing useful stuff. Maybe.
 
================================================================================

Andrew Adamson
bob at ucc.asn.au

|"If you can't beat them, join them, and then beat them."                |
| ---Peter's Laws                                                        |