I was waiting for you to say this, and I'm glad you did. I think this is what it boils down to for a lot of people, especially laypeople who are less informed about new technology than more informed parties, like the minds who developed the technology in question.

Fear can also derive from people getting comfortable with their current way of living, and not wanting to step out of that comfort zone with the introduction of something new and unfamiliar.

Something interesting to think about is this concept called "The Accident," which I read about in my favorite book of all time. "The Accident" is the idea that when things are invented, you also invent the accidents and potential damage that invention can bring. The invention of airplanes introduces the airplane crash. The advance of AI introduces the possibility of an entity far smarter than all of humanity combined that might decide humanity is more of a nuisance than something to aid, and therefore might become aggressive. Look up Roko's Basilisk if you're curious on the latter.

I think this might further add to the fear some people get. I don't think it's possible to invent something with absolutely no potential downsides.