[tech] Webcam archiving project complete

John Hodge tpg at ucc.asn.au
Mon Mar 29 20:15:24 AWST 2021


Hi All, here's an update on my recent UCC project:

Over the last few months (back to November) I've been working on packing 
our massive webcam archives into a more compact format (to both reduce 
the number of used inodes on /services, and make the backups a little 
happier).

The final option I used was low/no-compression .zip archives, but for 
future reference - video compression does not work well on 
already-compresses JPEG files (unfortunately...)

Changes done as part of this project
- webcam/archive.php updated to support reading images from Zip archives 
(and from videos, just in case)
   - Some slight profiling showed that the CPU cost of the archive search 
was minimal (in the order of 100ms)
- All cameras have had images from before 2020 compressed into archives, 
validated, and the originals deleted
- Any files that weren't normal files (i.e. symlinks, device files, ...) 
were deleted
   - A few years ago the server that hosted /services had a disk failure, 
and the metadata for the webcam archives was corrupted. These non-normal 
files are from then.

All in all, this has reduced the used inode count on /services from 
around 40 million to 9 million, which should help improve performance.


More information about the tech mailing list