Grab

2.3x faster using the Go plugin to replace Lua virtual machine

May 15, 2023Yonghao Hu
2.3x faster using the Go plugin to replace Lua virtual machine
SummaryAI, from the source post

This post details the migration of Talaria's data ingestion processing from Lua VMs to Go plugins. The Lua VM approach suffered from significant performance and memory overhead due to launching a new VM for each script. By leveraging Go plugins (compiled as shared libraries), Talaria achieved a 2.3x speedup and a 2.3x reduction in memory usage for custom data processing functions, bringing performance closer to native Go functions. The implementation involved creating a `plugin` loader that mirrors the `Handler` interface of the `LuaLoader`, allowing for dynamic loading and execution of Go functions from shared libraries.

More from the Data Lake Modernization with Apache Iceberg trail