Asexuality is neither inherently sexual nor inappropriate, even when used in conjunction with the SAM.
If someone were to tell you that they were biromantic and asexual, your first thought should not be about their sex life. You should not be thinking about whether or not that person gets intimate with others, nor how or with whom. If you do, that’s not a problem with asexuality—that’s a problem with you, and something that you need to address. Not to mention that people experience asexuality differently, and someone being asexual may affect their romantic attraction or interactions with others.
If an asexual person does not use the SAM and identifies solely as asexual, it’s no more inappropriate than someone saying they’re gay, bisexual, etc. It’s not okay for someone to come up to me and say “oh, you’re a lesbian? So you have sex with women?” simply because I’m not constantly emphasizing my romantic attraction to them. I should be able to talk about my orientation without people making assumptions about my sex life, just like asexuals should be able to talk about their orientation without people making assumptions about their sex life.
If your issue is with the term “asexual” while people of other orientations can call themselves lesbian, gay, bi, pan, ply, etc., then I have some good news for you: ace.