In order to establish the criteria that our comparison tool works with, we first created an evaluation model. We looked into meta analytical studies of website evaluation approaches. The studies analysed websites evaluation parameters based on user interaction between 1995 and 2011. We then grouped all the results from the studies that focused on commercial, educational and shopping websites, as these share more features with the core functions of a comparison website. The criteria we selected through this vetting process can be seen in the table below.
| Criteria | Description |
|---|---|
| UI | Evaluates how the UI of the relevant app performs in terms of eye strain, simplicity and cluttering. |
| Ease of Use | Evaluates how how quickly the app can be learnt and mastered and if there are any help topics to make a user's experience easier. |
| Mobile Integration | Evaluates how well the app performs across multiple platforms. |
| Cost/Payment Plans | Evaluates how a app service can be paid for (weekly, monthly, yearly) and what payment options are available. |
| Content Management | Evaluates what type of data the app can output and how compatible they are with generic data analysis tools. |
| Security/Ethics | Evaluates how user data is handled, what level of data security if offered and what legal ties are present in a subscription to the app. |
| Customer Support | Evaluates the customer support offered by the app. |
| Terms and Conditions | Evaluates what is hidden in the terms & conditions of the app. |
| Core Features | Evaluates the wealth and depth of the features that the app offers. |
To ensure that both features and functionality were assessed in each Twitter analytic tool, we devised a 5 question assessment evaluation form. 5 question were devised for each criteria to ensure equal weighting. The questions were written to guarantee a factual yes/no result. Each tool was then assessed and scored independently by four different users, to ensure that each question was answered correctly. The scores were then normalized. You can see the evaluation form below.
| Criteria | Questions |
|---|---|
| UI | 1. Is each section clearly labeled? |
| 2. Are all functionalities easy to reach? | |
| 3. Is the visual structure of the tool intuitive? | |
| 4. Is it fluid/non-sluggish? | |
| 5. Is the UI consistent across different browsers? | |
| Ease of Use | 1. Can basic functionality be learned in 30 minutes? |
| 2. Can the tool be mastered in less than 48 hours? | |
| 3. Are there help topics to guide the user through the tool? | |
| 4. Does the tool offer visual/textual feedback following an interaction? | |
| 5. Does it have clear instructions on how to operate its features? | |
| Mobile Integration | 1. Is there a phone optimised version of the site? |
| 2. Is there a tablet optimised version of the site? | |
| 3. Does it have a mobile application for iOS? | |
| 4. Does it have a mobile application for Android? | |
| 5. Are all features from the desktop platform available in mobile version? | |
| Cost/Payment Plans | 1. Is there a free version? |
| 2. Is there a refund policy for cancellation prior to notice agreement? | |
| 3. Are there different payment plans? | |
| 4. Can you customise a plan? | |
| 5. Can you get all functionalities with the cheapest plan? | |
| Content Management | 1. Can you create visualisations from your results? |
| 2. Can you export findings to PDF? | |
| 3. Can you export to CVS? | |
| 4. Can you export to XLS? | |
| 5. Does it have embeddable widgets? | |
| Security/Ethics | 1. Is you information shared with 3rd parties? |
| 2. Is your information confidential when shared? | |
| 3. Is a security/ethics policy available? | |
| 4. Is it easy to access these security/ethics policies? | |
| 5. Is it easy to understand the ethics/security policy? | |
| Customer Support | 1. Does it offer a customer support service? |
| 2. Does it have an email support service? | |
| 3. Does it have a phone support service? | |
| 4. Is there a customer support line 24/7? | |
| 5. Does the cheapest plan include all customer service options? | |
| Terms and Conditions | 1. Are there terms and conditions? |
| 2. Are the terms and conditions clear and concise? | |
| 3. Are the terms and conditions easily accessible across the site? | |
| 4. Do the terms and conditions exceed 3 pages? | |
| 5. Are there any hidden fees? | |
| Core Features | 1. Does it have more than 5 different features? |
| 2. Does it have community management? | |
| 3. Does it have hashtag analysis? | |
| 4. Does it have graphic visualisations? | |
| 5. Does it have a frequent users analysis? |
You can find the form results of each tool in the scores overview section.