Injection tuning

This section gives advice and examples on tuning the injection performance.

The following diagram shows the injection tuning process:

The injection is composed of three main steps:

  • Reading process – getting information from the input
  • Parsing processes – transforming input to usable objects
  • Injecting processes – injecting data in the database

The following settings are defined for the Datastore Runtime component using Administration:

  • Count of threads used to parse data during injection – defines the number of parsing processes
  • Count of threads used to insert data – defines the number of injection processes used to inject a file in the database.

Use a profile if the system has to manage different values of processes. A profile is defined in Administration and defined using an as parameter:

dstools inject --as MAIN_FLOW --collection AUDIT auditTrails.dat
MAIN_FLOW is the profile to use.

The tuning process is based on stack indicators displayed in log files. To generate logs with more or less details, change the dynamic setting Injection statistics modules in Administration.

Injection statistics modules: CONSOLE,CSV

Where:

  • CONSOLE: is used to display logs to the console output
  • CSV: is used to produce a file with injection statistics (Datastore/stats folder)
Note   The Injection statistics modules dynamic setting should not be defined in a production environment.

Sample of console output:

08:22:16 2339.92Mio /7162.09Mio 32% / 19400 r/s (i) 14010 r/s (m) Parse Ovf 1608419 -Stv 2263 Insert Ovf. 1915 -Stv. 3309 Mem: 46.28Mio /343.62Mio /3.46Gio

The sample output is composed of:

  • A timestamp: 08 :22 :16
  • Reading process position: 2339.92 Mio out of 7162.09 Mio (Mio is 1024*1024 bytes)
  • Reading process percentage: 32%
  • Instant injection speed (records per second): 19400 r/s
  • Average injection speed (records per second): 14010 r/s
  • The parsing stack information overflow (1608419) and Starvation (2263) information
  • The insert stack information overflow (1915) and Starvation (3309) information
  • Memory used by the injector tool: 46.26 Mio (Mio is 1024*1024 bytes)
  • Memory allocated for injector tool: 343.62 Mio (Mio is 1024*1024 bytes)
  • Memory allocation limit: 3.46 Gio (Gio is 1024*1024*1024 bytes)

Overflow and starvation indicators

Parsing and insert stacks have overflow and starvation indicators that follow these rules:

  • The overflow indicator increases when a process places an object into the stack but the stack is full.
  • The starvation indicator increases when a process tries to get an object from the stack but the stack is empty.

Overflow and starvation indicators are used to optimize the injection settings Parser thread count and Insertion thread count:

Indicator behavior Resources Action
Insert starvation (insert Stv) increasing during injection If there are CPU resources available on injector machine Increase the Parser thread count value
Insert starvation (insert Stv) increasing during injection if there are database resources (CPU and IO) available Increase Insertion thread count Parser thread count = Insertion thread count-1
  If there are no database resources available Decrease the Parser thread count value
Parse starvation ("parse Stv") increasing during injection   The input file reading speed is too low, store the input file on a faster disk

When the injector machine is an hyper-threaded system, CPU is available if global CPU usage is less than 50%.

Possible database issues

During injection, you may face some database issues pointed by database waits:

Database waits Remarks / actions
SQL Net - more data from client There are too many threads on injector machine: Decrease Insertion thread count and Parser thread count
User Wait I/O - db sequential read With the default architecture, index maintenance requires a lot of I/O. Moving to the one partition per sequence architecture solves this issue. See the Datastore Capacity Planning Guide for more details on the different types of architectures.

Post-injection actions

After performing a large injection, you must update the database statistics to ensure query performance.

Related Links