Modal

Product updates: Rollbacks, batching, sandbox tunnels & more

Sep 6, 2024
Product updates: Rollbacks, batching, sandbox tunnels & more
SummaryAI, from the source post

Introduces the `@batched` decorator which accumulates requests into dynamically-sized batches, allowing users to trade off latency for throughput and GPU utilization. Also adds the ability to forward TCP ports from `Sandbox` objects to the internet and `exec` into existing `Sandbox` objects, with a stated interest in improving Sandboxes for agentic use cases.

More from the Dynamic Batching trail