Removing Members. Unless the committee has been established by your governing documents and its members elected by the association, committees serve at the pleasure of the board. Accordingly, the board can remove committee members at will and without explanation.
Delegate Removal. The board can also delegate such authority to the committee chair or to the committee itself. If so, then the committee itself can vote someone off the committee.
Quasi-personnel. Committee staffing issues are quasi-personnel. As with paid staff members, if a board is faced with unproductive or misbehaving committee members, directors would be reluctant to discuss such matters in an open session for fear of embarrassing the committee member or, worse, setting themselves up for a defamation lawsuit. As a result, board members would be inhibited from freely discussing such problems if their discussions were in open session.
Disciplinary. Arguably, removal of committee members also falls under executive session for disciplinary actions. A noticed hearing with the committee member is not required since there are no monetary penalties imposed or loss of rights--committee members serve in an advisory capacity at the discretion of the board.
Privacy. Privacy is important because public admonishments of committee members would discourage other owners from volunteering for committee work. Hence, the discussion and vote on disciplining or terminating committee members should be done in executive session.