TechPrufer
Get started
GitLab
GitLab logoGitLab · Feature trail

Golang I/O Streams

1 post · Feb 2024

What this trail covers

GitLab's Gitaly, GitLab Shell, and Workhorse services, implemented in Go, efficiently handle terabytes of Git data transfer using streaming I/O. This involves composing Go's `io.Reader` and `io.Writer` interfaces to manage data chunks, stream content to HTTP responses, calculate multiple hashes simultaneously, combine multiple readers, and implement features like the Dependency Proxy by teeing data between readers and writers.

All GitLab trails
  1. Compose Readers and Writers in Golang applicationsThis post introduces and explains the practical application of Go's `io.Reader` and `io.Writer` interfaces and their composite functions (`io.Copy`, `io.MultiWriter`, `io.MultiReader`, `io.TeeReader`) within GitLab's Go-based services (Gitaly, GitLab Shell, Workhorse). It provides concrete examples of how these patterns are used for efficient data streaming, handling multiple destinations for data, combining input streams, and enabling simultaneous reading and writing operations, such as caching upstream images while serving them to users.