One of the reasons I hear the most from people as to why they don't want to start yoga is "I'm not flexible". I think this is very interesting for a couple of reasons. First reason being that yoga isn't all about stretching. Depending on the type of yoga class you go to, there is usually a fair amount of strengthening involved- sometimes more than stretching. The image that most people have of yoga is just a whole bunch of stretching and breathing. While the breathing is definitely true, a lot of the stretch comes from within strengthening poses. During a more challenging pose, usually we find stability or strength in one part of our bodies, while working to elongate other spaces. So while you will be getting a good stretch, you aren't expected to put your leg behind your head.
Yoga teachers want you to feel safe and secure in class. They understand that everyone is at a different place in their practice, and every individual body is completely different. The thought that a person would get kicked out of or mocked during a class because they aren't a contortionist is absurd. This notion is the exact opposite of what yoga is actually all about.
The last, and possibly the strongest point I'd like to get across is this. Would you not go to a swimming lesson because you don't know how to swim? Or skip out on a cooking class because you can't cook? Yoga class is meant to make you more limber and more strong. People go to class because they want guidance with yoga, not because they're already masters of it.
If yoga has caught your attention in any way, I strongly encourage you to try it. Just give it a chance and see how you feel afterwards. The way that your body will feel after class will give you hope that maybe being "inflexible" isn't enough to keep you away from yoga class forever.