Out of Character, or OOC, means exactly what it sounds like: characters not acting like themselves. Equally concerning are the more subtle changes authors make to characters. To some degree, this is unavoidable: even the best fanfiction writers can't replicate the canon characters exactly; only their creators can do that. What exactly constitutes being out of character is also subjective, depending both on how the fanfic writer has justified the changes and on the opinion of the reader.
More often than not, characters acting contrary to their normal natures is due to a Mary Sue warping the canon. Though it can happen to anyone, the Sue's romantic interest often suffers the worst, losing all of their faults and receiving major boosts to their abilities like some ungodly variation of Chuck Norris.
Some fanwriters insist that it is their right to warp characters OOC, on the grounds that it's their story and they can do what they wish with the characters. This is ignoring that what they're writing is fanfiction; it isn't really their story or their characters. There are also some who honestly don't realize that they're making the characters act OOC.