Put the specific rules first and the generic second. Once a rule is matched the others will not be evaluated (see manual below) so you must put the specific rules first. As rules change you may need to delete old rules to ensure that new rules are put in the proper order.

So, first add “allow from XXX port 1234”, later “deny 1234”.