You missed the point while drawing your circular argument.
Take what you said and replace fly with human. Wait here I’ll do it for you:
If a human lacks agency, you would be able to predict its movement given a sufficiently accurate set of information.
If it has agency, you could not.
Now tell me how you will acquire a sufficiently accurate set of information about a human and its environment to test your hypothesis.
Scary in what context? For the user? For you as the host? Something else I’m not thinking of?