“An intriguing open question is whether the LLM is actually using its internal model of reality to reason about that reality as it solves the robot navigation problem,” says Rinard. “While our results are consistent with the LLM using the model in this way, our experiments are not designed to answer this next question.”
The paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs” can be found here.
Abstract
We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifically, we train a Transformer model on a synthetic corpus of programs written in a domain-specific language for navigating 2D grid world environments. Each program in the corpus is preceded by a (partial) specification in the form of several input-output grid world states. Despite providing no further inductive biases, we find that a probing classifier is able to extract increasingly accurate representations of the unobserved, intermediate grid world states from the LM hidden states over the course of training, suggesting the LM acquires an emergent ability to interpret programs in the formal sense. We also develop a novel interventional baseline that enables us to disambiguate what is represented by the LM as opposed to learned by the probe. We anticipate that this technique may be generally applicable to a broad range of semantic probing experiments. In summary, this paper does not propose any new techniques for training LMs of code, but develops an experimental framework for and provides insights into the acquisition and representation of formal semantics in statistical models of code.
I mostly get what you’re saying, though I don’t have the requisite understanding to follow formal proofs, but if there is one thing I do know for certain, it’s that “understanding” is anthropomorphizing and shorthand for something that is very much not understanding in a human context at all.
I get that it can be hard to find the right words to explain a some of these emergent phenomena, but I think it’s misleading to use words that make AI appear to have a thought process akin to anything we could understand as such—at least in settings where folks might not understand the shorthand as such.
And maybe everyone here is aware of that, but it makes me uneasy, hence this comment to hopefully make that point.
The paper is kind of saying that as well. I added a quote to the post to help set the context a bit more. As I understand it, they’ve shown that an LLM contains a model of its “world” (training data) and that this model becomes a more meaningful map of that “world” the longer the model is trained. Notably, they haven’t shown that this model is actively employed when the LLM is generating text (robot commands in this case), only that it exists within the neural network and can be probed. And to be clear - its world is so dissimilar from ours, the form its understanding takes is likely to seem alien.
As someone who understands formal proofs, it’s completely misleading to conflate formalism with sketchy pedagogical theories (wtf).
Yes, terminology like “understands” is a choice outside of formalism that’s intentionally misleading for the sake of marketing/funding.
Genuine question: What evidence would make it seem likely to you that an AI “understands”? These papers are coming at an unyielding rate, so these conversations (regardless of the specifics) will continue. Do you have a test or threshold in mind?