2.3x faster using the Go plugin to replace Lua virtual machine
May 15, 2023Yonghao Hu

Summary
This post details the migration of Talaria's data ingestion processing from Lua VMs to Go plugins. The Lua VM approach suffered from significant performance and memory overhead due to launching a new VM for each script. By leveraging Go plugins (compiled as shared libraries), Talaria achieved a 2.3x speedup and a 2.3x reduction in memory usage for custom data processing functions, bringing performance closer to native Go functions. The implementation involved creating a `plugin` loader that mirrors the `Handler` interface of the `LuaLoader`, allowing for dynamic loading and execution of Go functions from shared libraries.