Clearly you should be much better than that. We do a lot of mixed fleet racing and feel we get on fine.
Is this a problem in all wind strengths? Or certain wave conditions? How old are your sails? You mention backstay… You getting enough tension when conditions warrant? How much backstay purchase do you have? Next time how about taking a halyard forward, tensioning it and taking a picture to share how much sag you’ve got.
Balance is important to pointing. Are you using enough main leach tension? And when you do, is the boat balanced? Are you sailing with max weight when called for?
Where are you and is there any chance you can get a local sailmaker to come out?