Obligations?

I have a question for colleagues out in the ether of the internets.  Let me preface this by saying that I try to be a good departmental citizen.  I do my part.

Say there's a job candidate coming.  In the past, the department has put out a sign up sheet, and we've signed up for a meal, and then we've had the meal with the candidate. In recent searches, however, the department chair has done a thing where zie has emailed to arrange the meals, basically, a "can you go to this meal on this date with the candidate."  And in the past, I've said yes and gone.

But it's never as convenient as when I can pick my own meal.  And one of the advantages of picking one's meals was that you could see who else was going, and hopefully sign up to eat with some folks whose company you enjoy, along with the candidate.

Here's the question.  I got the "have this meal with this candidate" email, and said yes.  But the time's not great.  And what's more important, the other faculty folks assigned to the meal aren't folks whose company I enjoy.  It's not that I hate them; I just don't enjoy their company, and don't generally socialize with them.

So, on a scale of 1 to 10, with 10 being unforgivable and horrible, how bad would it be if I emailed the chair and said that the meal assignment wouldn't work out for me?  (It's plenty early, so it wouldn't be a matter of causing impossible problems for the chair.)