Testing Methodology
Please click
here
to view or download a detailed introduction to Myce’s Enterprise Class Solid
State Storage (‘SSS’) Testing Methodology as a PDF.
Put briefly:
All testing is performed on an OakGate Technology test unit
We perform two sets of Performance Tests:
- A full set of the mandatory Storage Network Industry
Association’s (‘SNIA’) tests as specified in their Solid State Storage
Performance Test Specification Enterprise V1.0 – SNIA
SSS PTS Version 1.0. - A set of tests, known as the ‘Myce/Oakgate Full
Characterisation Test Set’, that provides readers with a fuller
characterisation of the solution.
We also review other important factors such as Power
Consumption, Data Reliability and Failover features.
A word about SNIA testing – before striking a partnership
with OakGate Technology I spent some time researching how I may implement SNIA
testing using freely available tools such as IOMeter and FIO. I arrived at a
conclusion that whilst it was theoretically possible it was impractical. The
reason for this is as without the automation offered by a test bench, such as
the OakGate Unit, the only way to meet the SSS PTS requirements is to run the
maximum number of test cycles and then to manually look back at the results to
determine when/if steady state has been achieved in the workload specific test
cycle, and then harvest the data from the qualifying Measurement Window – this
means that the test runs would always take a maximum elapsed time, and there
would be a great deal of human effort required to review, gather, and report
upon the data. I empathise with, acknowledge and respect the efforts of other
reviewers who endeavour to meet the SNIA’s principles in their testing – I am
privileged and thankful to be able to use a superb test bench which automates
the whole process and allows me to meet the SNIA’s specification in full.
Before we move on, let’s remind ourselves of some basics –
When reviewing the performance of an SSS solution there are
three basic metrics that we look at:
1. IOPS – the number of
Input/Output Operations per Second
2. Bandwidth – the number of
bytes transferred per second (usually measured in Megabytes per second, ‘MBs’)
3. Latency – the amount of time
each IO request will take to complete (usually, in the context of SSS
solutions, measured in Microseconds, which are millionths of a second).
It is true to say that IOPS and Bandwidth had all been
growing rapidly before the advent of SSS solutions, but Latency can only be significantly
decreased by eliminating mechanical devices, and thus Latency is the single
most important aspect that SSS solutions deliver to enhance performance.
Latency in a technical environment is synonymous with delay.
In the context of an SSS solution it is the amount of time between an IO
request being made, and when the request is serviced.
Bandwidth, also commonly referred to as ‘Throughput’, is the
amount of data that can be transferred from a storage device to a host, in a
given amount of time. In the context of SSS solutions it is typically measured
in Megabytes per second (MBs).
A great enterprise SSS solution
offers an effective balance of all three metrics. High IOPS and Bandwidth is
simply not enough if Latency (the delay in an IO operation) is too high. As we
will see in the test results presented below, as Latency increases IOPS will
inevitably decrease.
Queue Depth is the average amount
of IO requests outstanding. If you are running an application and the Average
Queue Depth is one or higher and CPU utilisation is low, then the application’s
performance is most probably suffering from a ‘Storage Bottleneck’.
Another important factor to
remember is that SSS performance is influenced by previous workloads, not just
the current workload, and especially by what has previously been written to the
drive. As specified in the SNIA SSS PTS the goal of all good Enterprise level
testing is to provide consistent circumstances, so that results can be compared
fairly across different SSS solutions – it is for this reason that all of our
tests start with a purge of the drive, so that it starts in a ‘Fresh Out of the
Box’ (FOB) state. Most tests then have a pre-conditioning phase where the
drive is put into a ‘Steady State’ before the test phase begins. Put briefly, a
‘Steady State’ is achieved when the performance of the drive no longer varies
over time and settles into a consistent level of performance for the workload
in hand. You can find a detailed explanation of ‘Steady State’ and how it is
determined in the SNIA tests in our Enterprise Testing Methodology paper, which
can be viewed or downloaded as a PDF by clicking here.
For interest, here are some
generally accepted assumptions that differentiate the use and therefore the
approach to testing Enterprise/Server and Consumer/Client SSS solutions:
Enterprise/Server SSS
assumptions:
- The drive is always full
- The drive is being accessed
100% of the time (i.e. the drive gets no idle time) - Failure is catastrophic for
many users
·
The Enterprise market chooses SSS
solutions based on their performance in steady state, and that steady state,
full, and worst case are not the same thing
Consumer/Client SSS
assumptions:
·
The drive typically has less than
50% of its user space occupied
·
The drive is accessed around 8
hours per day, 5 days per week, and typically data is written far less
frequently
·
Failure is catastrophic for a
single user
·
The consumer/client market
generally chooses SSS solutions based on their performance in the FOB state
Esther
Spanjer, Director, SSD Technical Marketing at Smart Storage Systems, said, ‘I
am happy to commend Myce for their high level of professionalism and
cooperation during the review process’, Ms. Spanjer added, ‘I wish them every
success in their partnership with OakGate Technology and their initiative to
provide authoritative performance reviews for the Enterprise Solid State
Storage market‘
Now let’s head to the next page, to look at the results
of our SNIA IOPS (Input/Output Operations per Second) Test…..
