Searches that are conducted for safety reasons must be reasonably based. At the police academy we used to teach search incident to detention, but have recently stopped using that vocabulary, as it was the reasonableness of the safety concerns and not the detention that were at issue at trial. Safety searches can actually be done without a detention, if the officer has legitimate safety concerns.
The leading case law on safety search is a 2014 Supreme Court of Canada decision R. v. MacDonald. ( available online). In that case a police officer was attending a noise complaint and the suspect who opened the door had something in his hands, which the officer believed might been a knife. When the person turned his back on the officer and didn’t show what was in his hands, the Supreme Court ruled the officer had legitimate safety concerns that met the standard of reasonable grounds, and justified his entry into the dwelling house and his control of the suspects hands.
The courts have consistently supported police who have legitimate reasonably based safety concerns.
Expect that the reasons for your fear will be questioned, as many police write or act as if a claim of safety concerns will satisfy the courts standard of reasonable grounds. Carefully consider and articulate the reasons for your concerns.