By being alive we cause destruction, if not actively, then passively. When a bug hits the window of your car when you're out for a drive, it's a form of destruction. No harm was intended, but it just happens. So how far can we go to avoid harm and reduce suffering?
It's hard to say. The first step most vegans take is to avoid consuming, using, or wearing things with animal ingredients or which used animals in the manufacturing or testing process. This is why vegans don't eat meat or fish, don't wear leather, fur, or wool, and use household cleaning products that weren't tested on animals. Even this approach has limitations, since animal ingredients can be hidden in so many common products and even in medicine and medical products as mentioned in this article from NPR.
Does it take the philosophy to far? I don't think so. When I started avoiding products with animal ingredients, I thought it was going to be hard. For many things I used, however, I was easily able to find vegan replacements. As I learned more about what was in products, it became simpler to spot ones that didn't support what I believed in, and it became less of a struggle to find ones that were OK with me. There were some things that took some searching to replace, but once I found a replacement, it just became normal to use it very quickly.
As more people adopted this philosophy and driven, in part, by celebrity and non-profit advocates, more vegan alternatives have become available for everyday things we use and need. Yes, there are some products today that are impossible to purchase vegan such as some prescription medicine and health products, but unless people seek alternatives it will never change. By pushing the envelope and demanding more products be made vegan, we will have alternatives for everything, but it doesn't happen by accepting that we can't live differently and more compassionately. Some people may call it extremism, but it's how the world changes for the better. There will always be early adopters of a lifestyle, who are then followed by the mainstream, which is how veganism has taken center stage.
Having said this, you don't have to refuse to go outside and walk around for fear of destroying something, or refuse a drug that may save your life because it caused suffering to develop. You can accomplish as much by encouraging others to change their habits or lifestyle and live more compassionately. Matt Ball of Vegan Outreach explains it nicely:
What we personally consume (especially at the margins) is almost irrelevant compared to what we can accomplish with thoughtful, honest advocacy for the animals. For example, influencing just one person to stop eating chickens and eggs — or even simply cutting back! — has an almost infinitely larger impact than if I avoid yet another obscure, miniscule animal product.