So far it seems like the ghosts only come after one person, the one they're looking for. In which case he could leave without trouble. But he's not doing that. He could also be wrong about that, and the ghost would care just as much about him as it does about Daenerys.

"No. It's Valyrian steel."