Improving machine learning iteration speed with faster application build and packaging

This post details improvements to the ML/AI development workflow by addressing slow build times and inefficient executable packaging. Key contributions include: implementing non-determinism mitigation in Buck2 and Remote Execution to ensure consistent build outputs, optimizing dependency management by identifying and removing unused code/dependencies (e.g., GPU code, unused Python modules), and introducing the Content Addressable Filesystem (CAF) for incremental packaging and distribution of Python executables. CAF utilizes a CAS daemon, P2P distribution via Owl, and Btrfs filesystem features (compression, COW) to optimize content fetching and deduplication. Future work includes lazy fetching and enforcing uniform revisions.