According to Gregory Benford, rendering the alien well is the holy grail of science fiction. I think he's right; in his Ocean of Night series, one character tells a family member that "The thing about aliens is, they're alien."
But there's a lot of science fiction around, and the sense of creepiness, other-ness, or sheer marvel that writers try to pack into their extraterrestrials starts to seem contrived when you've run across it one too many times. These efforts typically fall into one of the following patterns, starting with the least unlikely.
- X-Files approach: never filling in the gaps, making only eerie, creepy suggestions around the edges.
- The biochemist's approach: it's a lot of work to dream up an alien constitution that bears any scrutiny but it's rewarding to idea-oriented readers. A personal pet-peeve in speculative fiction is the use of an omniscient viewpoint, so that the reader knows what's going on but the characters don't, and you have to spend the waiting for them to catch up with you. Readers can sometimes be annoyed at this alien-rendering for the a similar reason; that is, even though the characters and reader are simultaneously discovering what makes the aliens tick, the author clearly knows all about it and you become impatient getting gradual fed bits and pieces of what is obviously from the start a coherent whole.
The Lovecraft approach - The alien's body/ship/language reeks of wrongness in a way that disturbs any unfortunate humans whose poor nervous systems dare look upon it!
The transcendental, or reverse Lovecraft approach - The aliens are superintelligent and hail from a benevolent civilization, so that the merest brush with them results in being healed, or becoming benevolent oneself. Much is made of tears being shed upon the enlightenment that comes from contact with higher planes of mathematical beauty, etc. etc.
The highly evolved earth animals approach (also known as intelligent gerbils, also known as lazy) - pick a species on Earth and extrapolate its beliefs, behaviors and civilization if it were to evolve to sentience. Larry Niven is allowed to get away with this with the kzin, but that might be about it. Note that there are plenty of really weird living things right here on this planet but this approach almost invariably uses a mammal-equivalent, and the ones that aren't mammals are at least tetrapods.
The ridge-headed alien approach - I forgive television series like Star Trek for doing this to stay within the constraints of a budget, but if you're writing a book where it costs you the same (zero) to make up an original alien or a bald-headed guy in a silver robe, and you take Baldy, shame on you!