Talk the Walk: Teaching AI systems to navigate New York through language

What this post added

Introduced the 'Talk the Walk' research task for embodied AI, using 360-degree images of New York City for navigation. Developed a novel attention mechanism called MASC (Masked Attention for Spatial Convolution) to improve the guide bot's ability to focus on the map and predict the tourist bot's location, achieving over twice the accuracy on the test set. Explored emergent communication protocols (continuous vectors and synthetic language) and compared their performance against natural language, finding synthetic communication to be more efficient but less human-understandable. Released baselines and the dataset for 'Talk the Walk' to the research community.