TechPrufer
Get started
Hudson River Trading
Hudson River Trading logoHudson River Trading · Feature trail

Python Codebase Dependency Analysis

1 post · Jun 2023

What this trail covers

HRT has developed a toolkit called 'Tangle Tools' to analyze the dependency graph of its large Python codebase. This system helps identify and refactor tangled dependencies, which cause performance issues with runtime imports and static analysis. The tools leverage Python's ast library, networkx for graph algorithms, and libcst for code transformations. This enables more efficient development iteration loops and reduces CPU waste in their distributed compute environment.

All Hudson River Trading trails
  1. Building a Dependency Graph of Our Python CodebaseThis post introduces 'Tangle Tools', a new system for analyzing Python codebase tangling. It details the problem of code tangling, its negative effects on import times and static analysis, and the approach to untangling through layered architecture. The post describes the workflow using Tangle Tools: finding unwanted dependencies, tracing import paths, refactoring imports with code transformations, and implementing dependency validation in CI. It highlights the use of open-source libraries like ast, concurrent.futures, networkx, and libcst.