Purge data

Decision Insight collects and generates a lot of data, saving all updates separately. Disk usage may grow indefinitely, which could impact performance.

Data is usually needed for only a recent period of time for specific entities, and older data must be discarded and removed from physical storage.

What to purge

The following functional categories are defined:

  • Transaction entities – May produce a lot of data which should have a rather short lifetime. All related data (instances/members values) should be removed at the same time. 
  • Stable entities – Have a very long lifetime and are not subject to a purge. Nonetheless, they may contain some indicators that produce a lot of data (like computing results). In this case, only members should be purged.

Following these definitions, a purge configuration is only available for entities and rhythmic indicators.

Glossary

The purge defines a few new terms:

  • Requested retention period - the time period during which the user wants to keep the data in the system after it is complete.
    • This is the requested value from the user, but may not be the one the system will really use.
    • It may be exported.
  • Applied retention period –  the real duration used by the system. When this duration is over, it means data is eligible to be removed from the system.
    • This is a calculated value based on rules.
    • It is not exported.

Coherency rules

Applied retention period values respect the following rules:

  • Only rhythmic members may have a specific requested retention period.
  • If a requested retention period is set on an entity, its value is propagated to all the members, keys and specializing elements, overriding specific rhythmic requested retention period if any.
  • Top level entities always override specializing values.
  • If an entity has multi-inheritance, the lowest retention period value from all of its parents is applied.

Purge data

To purge data from the deployment, on the main menu, click the Configuration   icon. On the left menu, click Purge in the Model section.

General view

From the purge screen, you can set the purge configuration for a specific entity or indicator, or do a mass selection to apply purge configuration to a selection of elements.

The following screen shows the Purge screen in Decision Insight.

List of screen elements:

  1. List of selectable elements
  2. Rhythm of displayed indicator
  3. Editable configuration 
  4. Mass edition region when elements are selected
  5. Filters

    You can filter the displayed entities and attributes by the following categories:

    • Item kind: Entity or Attribute
    • Rhythm: all available application rhythms
    • Space: all available application spaces
    • Entity type: if an entity is of type configuration, all attributes will be displayed
    • Retention period: combination of requested and applied retention period

    When you select All, no filter is applied to the corresponding category.

    If multiple filters are set, the displayed lines are filtered using the AND operator between the categories.

  6. Save and Cancel buttons. No changes are applied until you click the Save button. Once you click Save, the screen is then refreshed to display what was really saved. The Cancel button reverts all unsaved modifications.

As soon as a duration is saved in the database, data can no longer be requested past the retention period time, even if a larger duration is specified afterward.


The user interface may display elements with different typographies:

  1. standard blue font – requested and applied values are the same.
  2. bold font – a pending modification that will be persisted once you click Save. The displayed value may not be the one applied.
  3. standard blue font with a warning icon – requested and applied values are not the same. The warning tooltip displays both values.

Update the retention period of an element

To update the duration of any element, click the Retention period cell. A combo box is displayed with all possible duration options that the element can use.

When you select a new duration, the updated value is displayed as bolded.Your retention period duration is sent to the server only once you click the Save button.

Mass update

Until you select an element in the list of entities and attributes, the mass edition header is disabled.

To select all elements in the list, select the checkbox next to the Entities and attributes header.

As soon as at least one element is selected, you can select a new duration and apply it to your selection:

When you click the Apply button, all selected elements have their duration updated, and the selection is cleared.

The Unlimited option means that purge is deactivated for the element. No data will be deleted.

Add a new retention period duration

By default, the drop-down box for the retention period selection displays all the durations currently used by the elements in your list. You can quickly set a value by reusing an existing duration.

To add a new duration:

  1. In the drop down box, select the click the New retention period... option.  A pop-up is displayed.
  2. Specify a new retention period.
  3. Click Create.

Configuration

A purge mechanism can be run periodically to delete outdated data. Please note that:

  • If data is referenced by a locked checkpoint, it will never be deleted.
  • If data is referenced by a checkpoint (not locked), it will be physically deleted as soon as no more references are found.

Important: In a clustered deployment, you define what to purge from the graphic interface and the purge frequency at the node level using the platform.properties file. Ideally, all the nodes in a cluster should have their platform.properties file configured the same way. For information about how to schedule a purge process, see Node settings.

File disk purge

The physical purge process, i.e. shrinking or deleting files on disk, is run automatically once a day (during server night hours at 01:00). It uses the configured retention periods.
New (smaller) purged files are created as a copy of the existing files. This temporarily creates a small increase in disk usage. Once the unpurged versions of the files are not used anymore, they are deleted, which frees up the disk space.

For the purged files to be deleted, you must wait at least 3 new checkpoints after the physical purge process ~2h (with default configuration). Do not create any more checkpoints before the physical purge process in order to free disk space.

Manual disk purge

To launch a physical purge manually, see Shell commands reference.

The following example is only for testing purpose. Do not test it in production environment.

From a running node containing some data:

  1. Set the maximum number of checkpoint to 1 using setMaxSpareCheckpoints 0.   Only one checkpoint is kept.
  2. Note the disk size of ..\var\data\titanium-temporal\partition-0
  3. Set the retention period for the entities you want to purge. 
    Tip: On a testing environment, choose a small retention period to see substantial changes to the size of the databases. 
  4. Execute the purgeStorage  gogo command to execute physical purge. 
  5. Generate a new checkpoint using createcheckpoint "checkpoint for purge test"
  6. Check the checkpoint creation in ...\var\data\calcium\checkpoint and ...\var\data\titanium-temporal\configuration\checkpoint.
  7. Copy the checkpoint number. This corresponds to the checkpoint name without tt.
  8. Unlock the checkpoint created in the previous step: unlockCheckpoint <checkpoint number of previous step>.
  9. Check the sstable files in ...\var\data\titanium-temporal\configuration\sstable to verify that all previous days are purged (but not the current one).
    1. - p-<period begin VT (0 padded)>_<yyyy-MM- dd>-<a/r> => you'll need to convert period begin VT (0 padded) to human readable value. 
    2. Compare the file size before and after purge.
    3. Check the suffix version (if no suffix = no purge). The suffix is incremented after each purge.
  10. Note the disk size of ...\var\data\titanium-temporal\partition-0 and compare it the one of Step one The last one should be smaller than the one of Step 3.



Related Links