For software to be trustable the public needs be able to see what it does (not necessarily as an individual but collectively). Normally that means we need the source code but “AI” can mean a system like an artificial neural network - source code isn’t enough to understand what it is actually doing (even with the training data).