If empty, the follower is pressing up on the slide release that is somehow causing the slide release to engage the grooves in the trigger pin.
If that’s it, I am guessing some sort of damage to the slide release or it’s spring(and/or the trigger is pressing on it) that is causing it to work almost opposite of how it is supposed to since lifting and wiggling the slide release is part of the process of pushing the trigger pin out.
Check for damage or burrs on the slide release and trigger where they interact with the trigger pin.
If it is happening with a loaded mag I am out of guesses.