The hotel manager canceled my room reservation

Two weeks ago, I booked 2 rooms in a hotel for 5 days in San Diego through Priceline.I checked in one room for two adults uder my name in the afternoon. At night, when my wife checked in the other room under her name for 2 adults and 2 kids. The manager got mad for un-known reasons. He said there were too many people and required us "leave my property immediatly" otherwise he "will call police". He did not give us any other choices. We had to leave at about 10:00 at night to find the other hotel.

Did they have the right to cancel my contract with Priceline for this weird reason (two queen beds in each room)? The manager really messed up the vacation of my family.