There are other workers from other castle looting in my castle how to prevent that it shows in the..

The icon in the watchtower shows if other players have explorers around your city. You cannot prevent this from happening and it does not affect you in any way.

However if you tap on the icon and to locate them and then tap on the works that are exploring you may have the option to Snatch their resources. Some workers cannot be robbed in this way but some can and it will let you grab some extras from them. Be aware that they will be informed of this so don't do it if you fear any reprisals from them.