I agree. If using a certain method or technique made our work more complex than necessary, we should discard it and find another way to achieve the same result. All of my teams never actually used story points. We'd rather slice the stories into a size that can be done in one sprint. In the end, that's how we estimate, i.e. what can be done in each sprint. The positive side is that it is much simpler and understandable by all parties involved.

Simplicity is essential and yes, we can be Agile (and use Scrum) without using story points.