To add a user to your channel’s access list, use:
/msg X ADDUSER #channel username level
To remove a user from your channel’s access list, use:
/msg X REMUSER #channel username
Both commands require you to have an access level of 400 or more. You can only add users with an access level lower than your own, and cannot remove users with higher level access than you.