1，要回定金我们有没有道理？
Your right is defined by (1) the initial agreement (2) your state's consumer protection law. You need someone to look at your initial agreement to see under what condition you can get your deposit back, and see if their conduct meets the condition. If the conditoin is too strict, you might seek to void the contract under consumer protection law.
2，如果该要，怎么操作？只能打官司吗？
Small claim court would be approporite.
3，我们认了，定金不要了，他们会不会说是我违约，向我们要违约金？
Again, you need to see your initial agreement to see if the deposit serves as liquidated damage (违约金). If so, you would not be liable for anything else.
The worst case would be, they said "材料已经下了订单", assuming the material is 'custom made', and cannot be used on other project, you might be liable for the cost of those materials. But again, I think the possibility of them suing you is remote.
