Reliability - Look for a high uptime percentage. 99% uptime is the minimum acceptable standard. Anything below 99% is not worth your time.
Disk Space - How much disk space do you require? How much disk space does the web host offer? Make sure the webhost package meets your needs.
Bandwidth - This factor is usually overlooked until the bandwidth limit is reached. If you go over your alloted bandwidth, you will get a hefty bill. If you are running a site with lots of downloads, or a site with lots of images such at a photo post site, your bandwidth will be a lot higher than a site with mainly text.
CGI Bin - This should be a standard with most hosting companies. If you do not have access to your own CGI Bin, you cannot install scripts and programs.
Perl - If you do not have Perl, you cannot run Perl Programs. Many good scripts that you buy or get for free are written in Perl. In my opinion, it would be a limitation not to have the latest version of Perl installed. MySQL - What scripts are you going to run? Some programs require a MySQL database to run. I had this problem with a particular webhost. They wanted to charge me $10.00 per month for a MySQL database. WOW. I just took out a new plan with another hosting company for under 10 bucks a month and 5 MySQL Databases are included in the price. Say no more.
PHP - Many new scripts and programs require PHP support. Again, this feature can be a $10 extra but most good hosts offer it standard. Imagine if you do not have PHP support but the latest new FREE toplist, blogging script requires PHP. Again, this feature is almost standard these days.
SSI - Server Side Includes are great if you want to spend minimal time updating your site. Here's how it works. This page contains SSI's. The left and right nav menus and the little search engine in the left nav menu are "included" with an SSI command in the master page template. If I need to update one of the nav links, there is no need to edit all the pages that contain the menus. I just edit the one page which contains the menu HTML and upload it to my site. It's awsome.
Domain Name - What's the point in buying web space if you cannot point your own domain name to the site? Select a host that allows you to point your own domain toward your site. This way you are in control of your domain name if you ever change hosts.
FTP Access - Unlimited and unrestricted FTP access to your site is essential. You will need an FTP program to transfer files.
Control Panel - Make sure your host offers Cpanel. It's the most popular and most user friendly control panel interface. In my experience, I have always been disappointed with the quality of the control panel when a demo was not available.
Statistics Program - A good stats program is handy if you want to keep an eye on how many visitors you are getting, where they are comming from, refferers, top entry and exit pages and so on. Some companies offer excellent stats programs as standard while others offer basic stats but you have to pay extra for advanced statisics.