How do you pass Into the Bandits Den quest?

I'm doing the quest shown above and it won't allow the bandit guy in the secret room with the push button. I walk in and he just stays behind. I tried talking to him and that didn't do anything, so did I miss something or is it a bug?