I explicitly remember being issued permethrin treated ACUs that I wore during summer training in Alaska. Same with the issued mosquito bed nets.

I have no idea about any long term effects of it. However, day to day use won't show any ill effects...long term, no clue. To be honest, I just use sprays versus treated clothes while hiking and camping, so if the permethrin concerns you, then it may just be worth it to avoid them altogether and keep doing what you're doing.

I personally wouldn't think twice about wearing it once in a while.