IDG Answers is a community of experts who are passionate about technology. Ask a question or answer one below.
He actually does not have to physically have your phone. All he has to do is call your number and if he has hacking software, it will hack your phone.
If you want to know more about it, do a google search on phone hacking or "how to hack a phone". Sadly, I believe I was hacked. Get away from that boss. If you are right, he/she is a sociopath. Very dangerous.
It is almost impossible for your phone to have had spyware installed by your boss unless he had it in his physical possession or had access to your Google Play/Android Market account. Have you noticed any unusual behavior on your phone? Does it make weird sounds, or has battery life noticeably decreased? Has it started freezing when it didn't before? I suggest you check your applications, and see if there is an innocuous sounding app called "SmartPhone" with a generic Android app icon. If that is on there, then you can be sure that there is spyware installed on your device. I don't know how you would prove who put it there though, and I suggest you be very careful about making accusations that you cannot prove, even if you have very strong suspicions.
To be certain that there is absolutely sure there is nothing on your phone is to do a factory reset.
I just realized that I assumed your phone is Android, sorry if it isn't. I don't know as much about iPhone spyware, so I can't help you there. Good luck.