Mobile Network Operators – Who’s the Best in the UK?
Here in the UK we have four Mobile Network Operators EE, O2, Vodafone and Hutchison 3G (Three Mobile). Between these four MNOs there are approximately 83.1 million subscribers making up the UK mobile usage market.
With these different choices in airtime provider who should the end-user go for? It’s a prickly question, ask someone and each time you’ll get a different answer with different people vouching for their favoured network.
Usually this is based on length of time with the network, their experiences with customer services and the deals that the customer is being offered – naturally all quite biased.
So isn’t it about time someone looked into the question of ‘just who is providing the better service’?
Well it appears that P3 Communications, the international leader in mobile network benchmarking has been asking those very same questions and decided to take a scientific and methodical approach to finding out the answer.
According to P3 Communications, “The UK is one of the largest and most interesting mobile communications markets in Europe since it is moving very quickly at the forefront of trialling and deploying some of the newest LTE technologies.“
P3 has been running similar annual mobile network benchmarks in Germany since 2002 and in Austria and Switzerland since 2009 – the well known "Connect Test".
Two specially adapted cars were used Each vehicle was fitted with P3’s custom-built proprietary high-end measurement systems. There was one voice and one data test smartphone per network operator – underneath a fitted roof top box there was a set of external antennas to ensure proper test conditions.
All of the smartphones used were fitted with 12dB attenuation in the antennas to help reflect the balance of real-world usage; such as users occasionally moving into buildings etc.
As the test measured across a range of signal frequencies the choice of smartphone for taking the 2014 UK measurements was of great importance, this is why P3 decided to use the Samsung Galaxy LTE Cat 4, the device supports downloads of up to 150Mbit/s.
EE, O2, Three and Vodafone provided the SIM cards themselves for the testing and ensured that any fair-usage policies would not interfere with the test results. These policies are often used to control the data download speeds for customers with very high recorded data usage. P3 test staff, however, went out to anonymously buy SIM cards from typical retail outlets to ensure that their control SIM cards would match those made commercially available to consumers.
Voice Tests - All of the voice test calls carried out were placed mobile to mobile and were set-up to be taken from each P3 test car to the other P3 car. During the calls the cars were moving around in cities, waiting at hot spots, or transferring between cities.
The speech quality of the calls was evaluated using Perceptual Objective Listening Quality Analysis(POLQA) wideband algorithm. This has become an important measurement standard for next-generation networks offering high-definition voice services, support for VoIP transmissions and the latest voice coding technologies.
All test cases were undertaken using the LTE preferred mode, which forced the networks to use CSFB while the devices were in a 4G capable area – when in 2G only or 3G capable coverage areas; calls were undertaken as usual. To give a ‘real-world’ feel to the test the devices were also handling data traffic at the same time – by receiving emails as default.
There were several key performance metrics being monitored including call setup times, speech quality, call success ratios and sustainable connections. There is a distinct difference between the last two metrics - call success ratios and sustainable connections. Call success ratio looks at the proportion of calls set-up successfully and do not cut off.
Sustainable connection is a measurement of calls that do not suffer a drop in speech quality. The call cannot drop below a certain threshold where in user terms the call quality is so unsuitable that they need to hang up due to either noise on the line or where there are problems hearing the other person.
Data measurements - The data measurements were taken from a variety of different data-specific tests; included were HTTP downloads and uploads, live web browsing, static web browsing using a defined standard web page, and watching YouTube videos streaming in both SD and HD (360p for SD and 720p for HD).
There were several web pages chosen to test the data connections from amongst the most popular sites within the UK, the following web pages were selected:
- The Telegraph
As one final test there was a ‘stress test’, whereby as much data possible was pushed over the networks within a 10 second period.. Several key metrics were chosen for the data measurements including success ratios and session times, upload and download speeds. There was another test for YouTube videos which was the delay length for video start-up and the percentage of playback samples without interruption.
You’d think that an operation of this scale would take a lot of planning but P3 communications managed to get the benchmarking tests underway in a surprisingly short amount of time. The original scoping and planning work was undertaken in June 2014 with the network operators informed of the upcoming tests in July. P3 defined all test-parameters in September, testing began in October. All the MNOs were notified and advised to inform P3 of any outages which might affect the results. The testing concluded in November and the results were published on the 25th November 2014.
In total the P3 testing vehicles covered more than 3,500 miles. Between all services on all four networks, more than 160,000 data samples were collected. Roughly 16,000 voice calls were made, with more than 125,000 speech samples used. In total more than 380 Gigabytes of data was transferred.
Results and conclusions
Across the various test benchmarks a total score of 500 is available - the score reflects the degree of fulfilment regarding the maximum achievable performance for voice telephony and data.
The overall score combined results from all benchmarks. Based upon results of all of the tests there was one clear winner; EE achieved the highest overall score with a significant lead over the second placed network Three.
O2 and Vodafone are pretty much on par with each other although O2 has a very slight edge.
The P3 Mobile Benchmark was specifically designed to push the limits of the most advanced mobile network technology currently available. All-in-all the test revealed some very well served geographical areas, but some companies' services showed plenty of room for improvement. The good news for UK customer is that the newest technology potentially provides a means to obtaining better capabilities than the services currently being offered.