Benchmarks


Introduction

It is important to be able to define the hardware configuration needed to support the workload generated by the system that Embedded Analytics for API will monitor.

We made some benchmarks on different configurations and scenarios. This document describes the benchmarks conducted and the results observed.

This article can help the customer to select a hardware configuration for Embedded Analytics for API.


Benchmark scenarios

The purpose of this benchmark is to monitor Decision Insight behaviors :

    • Response time, 
    • CPU Usage, 
    • Memory Usage 
    • Hard disk Usage

We ran several test campaign to determine the limits of a Decision Insight server configuration. Some tests inject increasingly higher number of transactions and some others inject a constant throughput but over a long period of time


Test environment configuration and assumptions

The performance of the Embedded Analytics for API depends not only on the number of transactions injected, but also on the customer context as configured in API Gateway manager.

For these benchmarks, we have the following configuration:

  • 10 Applications
  • 200 Application methods
  • 10 client Application
  • 10 organizations


Architecture Configuration

To do these benchmarks, we use some external tools to simulate API calls and monitor the resources useage.

The architecture layers is:

   

    • Testing tool layer
    • API gateway layer
    • Decision Insight Architecture layer
    • Monitoring layer


The environment for benchmark used 3 servers:
 

  • Jmeter server to generate API calls
  • API Gateway server composed by an Apache load balancer, 2 API gateway and a Pet Store sample
  • Decision Insight Server with benchmark monitoring tools (grafana, InfluxDB, cAdvisor) for this server 


Material configurations

We run these benchmarks on the following configurations:


  • On Virtual Machine:


Virtual CPU type used

CPU model name Intel Xeon® CPU X5650 @ 2.67GHz




Configuration

Docker
Corresponding Server

CPU limit

RAM limit


vCPU RAM Disk
configuration 1 2 4 GiB
2 6GiB 100 GiB
configuration 2 4 4 GiB
4 6GiB 100 GiB
configuration 3 2 6 GiB
2 8GiB 100 GiB
configuration 4 4 6 GiB
4 8GiB 100 GiB
  • On Amazon:


Instance type used
m4.2xlarge: 8vCPU 32GiB
CPU model name Intel Xeon® E5-2686 v4 (Broadwell) processors @ 2.3GHz
  or   Intel Xeon® E5-2676 v3 (Haswell) processors @ 2.3GHz
  • EBS-optimized by default at no additional cost
  • Support for Enhanced Networking

Balance of compute, memory, and network resources


Configuration

Docker
Corresponding Server

CPU limit

RAM limit


vCPU RAM Disk
configuration 11 4 8 GiB
4 10 GiB 300 GiB
configuration 12 4 10 GiB
4 12 GiB 300 GiB

Benchmark types

Gradual throughput increase test of API Gateway


Rhythm Request per second Cumulative data
t 1 0
t+10m 10 600
t+20m 50

6,600

t+30m 100 36,600
t+40m 200 96,600
t+50m 500 216,600
t+60m 500 516,600

  

Gradual throughput to the API Gateway for a longer period of time

example: 3 days with a throughput at 500 tr/sec:


Rhythm Request per second Cumulative data
t 1 0
t+1m 10 60
t+2m 50 660
t+7m 100 15,660
t+17m 200 75,660
t+27m 500

195,660

t+3days 500 130000000

  


Observations / Comparison between results tests

Disk

Decision Insight has computations at different rhythms and various checkpoints create some disk usage peak.

It's important to consider this temporary use of extra data space.

Chart to represent disk used in function of data received

Basic rule to define space

For this bench environment configurations and hypothesis , the rule for Embedded Analytics for API Gateway is a function of:

  • the quantity of data computed in time
  • the retention period of the data
  • the average size of  single messages sends by filebeat and received by Decision Insight


This formula can define the closest disk space without margin:

50GiB + <Throughput in transaction per seconds> x 60 x 60 x 24 x <number of days> x (< transaction size in KiB> / (1024 x 1024)) x 1,15

  • Throughput is the number of transaction send per second by Filebeat and receive in Decision Insight.
  • Number of days that you want to keep informations in data base.
  • Size of a transaction. (in our cases the average message size is 1,8KiB)

This formula is a based to determine the sizing space for a server.

Don't use this formula to determine hard disk size for Decision Insight server.

Please use  Decision Insight Embedded Analytics API Gateway sizing guide to size the server with margin.

CPU

The number of CPU is used by Decision Insight to:

  • integrate data
  • compute data at different rhythm
  • retrieve data requested by dashboard
  • run the webserver

During bench, the number of CPU used to integrate data is:

Throughput tr/sec CPU usage
<=50 10%
100 40%
200 80%
500 120%

You can add intermittent computing that can use more than one CPU.

Those informations is used to determine the sizing of CPU for a Decision Insight server.

Don't use this rule to determine hard disk size for Decision Insight server.

Please use  Decision Insight Embedded Analytics API Gateway sizing guide to size the CPU of a Decision Insight server.

Memory

The Java memory used by Decision Insight for the API Gateway embedded analytics application is important to compute data faster.

During bench, the maximum size of JVM used to integrate data is:

Throughput tr/sec JVM used*
<=100 4 Go is sufficient.
200 6 Go at least.
500 At the end of the long test with the configuration 12 (8GiB) the 8 GiB of Java memory is fully used and  the configuration 11 (6GiB) crash when Decision Insight compute data. The memory is completely used.

*caveat: the tests have not taken into consideration the impact of increasing dimensions on the memory used by the JVM.

For high throughput (high thoughput during longer periods test), you can see that there is an overhead on the JVM. This overhead is less than 2 GiB.

Please use  Decision Insight Embedded Analytics API Gateway sizing guide to size the memory of a Decision Insight server.

 


Conclusion

To size a Decision Insight in the specific context of the Embedded Analytics monitoring application, the principal parameter to consider is the throughput of messages send by the API Gateway server.
This factor determines the required server hardware. Therefore, if the throughput change, you have to reconsider your server configuration.

The server configuration parameters are :

  • The memory : to be more efficient Decision Insight need memory,
  • The CPU : to integrate data, to compute for each rhythm all informations received and to compute informations displayed on dashboard,
  • Disk space : data saved on disk , for information extracted from API Gateway logs and aggregates and indicators computed using this data.

See Decision Insight Embedded Analytics API Gateway sizing guide to determine your server configuration


Related Links