The platform should use cookies so the visitor will see the exact same page on each consecutive visit. There shouldn't be any type of counter or display evident on the page being tested. All of the reporting should be done in the background. Is the platform easy to install? Particularly if you're using a software program (as opposed to a hosted service), you want to pay attention to the system requirements. If you're a non-techie entrepreneur, you'll probably want to stay away from a software that requires a SQL server on your own PC.
On the other hand, there are some great options that rely only on a MySql database on your web server, and, since you'd be interacting with a web interface, you never have to touch the database at all. Is the platform easy to implement? Most split testing platforms require you to make some type of change to your web page - and it's usually in the form of adding a piece of coding to the page header and/or body tag. Anything much more than this might get too overwhelming, and you run the risk of abandoning your split testing efforts before you really get them off the ground. Look for a platform that is easy to implement and easy to use once it's up and running. Is the data easily accessible? There's no point in running split tests if you can't quickly and easily see the results. Look for a platform that will provide visual representation of your data, such as bar charts or line graphs or pie charts.
If you're a techie, you might also want to consider a platform that provides access to the raw data so you can manipulate it and use it elsewhere. The big benefit of split testing comes from what you do with the results! What type of testing does the platform allow? If you're brand new to split testing, a simple A/B platform might be all that you need. But if you have a high traffic website, if your main income is from the internet, and/or if you're looking for data on multiple variables, you'll want to consider a split testing platform that will run multivariate testing. The price range for software and services varies greatly, as does the time commitment involved. But, since split testing has the ability to improve website conversions and increase your bottom line, it should definitely be seen as an investment in your business!