[tech] Webcam archiving project complete
John Hodge
tpg at ucc.asn.au
Mon Mar 29 20:15:24 AWST 2021
Hi All, here's an update on my recent UCC project:
Over the last few months (back to November) I've been working on packing
our massive webcam archives into a more compact format (to both reduce
the number of used inodes on /services, and make the backups a little
happier).
The final option I used was low/no-compression .zip archives, but for
future reference - video compression does not work well on
already-compresses JPEG files (unfortunately...)
Changes done as part of this project
- webcam/archive.php updated to support reading images from Zip archives
(and from videos, just in case)
- Some slight profiling showed that the CPU cost of the archive search
was minimal (in the order of 100ms)
- All cameras have had images from before 2020 compressed into archives,
validated, and the originals deleted
- Any files that weren't normal files (i.e. symlinks, device files, ...)
were deleted
- A few years ago the server that hosted /services had a disk failure,
and the metadata for the webcam archives was corrupted. These non-normal
files are from then.
All in all, this has reduced the used inode count on /services from
around 40 million to 9 million, which should help improve performance.
More information about the tech
mailing list