sysupgrade error reference

This section describes example warnings and errors generated by the sysupgrade script and provides recommended solutions.

Export command errors and warnings

This section describes example errors and warnings that the sysupgrade export command generates when exporting the data from the old API Gateway installation.

Errors and warnings are logged to the following file:

Axway-7.5.3/apigateway/export/bin/out/logs/export.log

Check for customizations to jvm.xml and service.xml

Every API Gateway system stores JVM settings in VDISTDIR/system/conf/jvm.xml. Configuration settings for individual API Gateways are stored in VINSTDIR/conf/service.xml (for example, apigateway/groups/group-2/instance-1/conf/service.xml). If you have modified these files in the old installation, the changes must be manually copied to the corresponding files on the new installation. The export step logs an informational message to inform you of this.

Each individual API Gateway instance can also have its own jvm.xml file at VINSTDIR/conf/jvm.xml. This file is not present by default, but can be created by users. Instance-specific jvm.xml files are migrated to the new installation as part of the upgrade process. The export step logs a warning to inform you to review these files and verify if they are still needed on the new installation. If not, remove them from the export directory before proceeding with the upgrade.

The following is an example of output from the export log file:

[System Config] : OK
[INFORMATIONAL] messages from [System Config] : 
Not migrating the following instance-specific service.xml files:
/opt/Axway-7.4.1/apigateway/groups/group-2/instance-1/conf/service.xml
If you have modified any of these files, you should copy your changes to the new system after upgrade is complete.
Not migrating apigateway/system/conf/jvm.xml. This is specific to the old Gateway version - a new jvm.xml 
will be present on the new system. If you have customized this file, you should manually copy your modifications to the new system.
[WARNING] messages from [System Config] :
WARN: Migrating the following instance-specific jvm.xml files from the old system: 
WARN:     /opt/Axway-7.4.1/apigateway/groups/group-2/instance-1/conf/jvm.xml
WARN:     These jvm.xml files are not part of the standard Gateway installation, and must have been added for 
customization. Please review these files and verify that they are still needed on the new system. If not, they 
should be removed from '/home/neil/dev/apigateway/upgrade/bin/out/export/sysconfig' before proceeding with upgrade.

Failure to export data from pre-7.2 installations

Data export can sometimes fail when attempting to upgrade pre-7.2 installations. The export step logs the following message describing how to proceed:

[KPS] : FAIL
[ERROR] messages from [KPS] :
ERROR: Failed to export KPS data from pre-7.2.0 system. Solution:
ERROR: (1) Create file '/opt/Axway-7.1.0/apiserver/conf/jvm.xml' with the following contents:
<configurationfragment><classdir name="$VDISTDIR/upgrade/legacy/7.1.x/"/></configurationfragment>
ERROR: (2) Rerun sysupgrade export.

Cassandra configuration

When upgrading from API Gateway versions earlier than 7.5.1, the export step checks if there are multiple API Gateways set up with one or more embedded Cassandra servers configured.

If there is only one embedded Cassandra server, the Cassandra configuration summary section simply says:

[Cassandra] : OK

If there are more than one embedded Cassandra servers and therefore more than one cassandra.yaml files present, the export step logs a summary similar to the following:

[Node Manager Entity Store Configuration] : OK
[Cassandra] : OK
[INFORMATIONAL] messages from [Cassandra] :
In the event of KPS Export errors, confirm that the following files are correctly configured :
/opt/Axway-7.4.1/apigateway/groups/group-3/instance-2/conf/kps/cassandra/cassandra.yaml
/opt/Axway-7.4.1/apigateway/groups/group-3/instance-3/conf/kps/cassandra/cassandra.yaml
/opt/Axway-7.4.1/apigateway/groups/group-2/instance-1/conf/kps/cassandra/cassandra.yaml

The reason for this message is that there is typically only one API Gateway acting as an embedded Cassandra server. However, you can configure Cassandra to use multiple network cards on the same machine. In this case, there are multiple embedded Cassandra servers (and therefore multiple cassandra.yaml files present), and sysupgrade logs the above message.

In addition, an incorrect Cassandra configuration might also cause KPS export errors.

The solution in all cases is to ensure that Cassandra is configured correctly in the old installation. For more information, see Check that embedded Apache Cassandra is configured correctly in the old installation.

Inconsistent groups

If your old installation contains any inconsistent groups, the export step logs the following message:

Group 'TestGroup' (group-2) is inconsistent. Upgrade requires all groups to be consistent.

To resolve this issue, ensure that all API Gateways in the group have the same configuration deployed.

Upgrade command errors and warnings

This section describes example errors and warnings that the sysupgrade upgrade command generates when validating the data exported from the old API Gateway installation.

Errors and warnings are logged to the following file:

Axway-7.5.3/apigateway/upgrade/bin/out/logs/upgrade.log

Admin Node Manager host name is invalid

If the Admin Node Manager has not been configured correctly, the upgrade step logs the following errors:

ERROR: Host 'XXX' is not a valid host in the topology. Retry with --anm_host <hostname>, where hostname is one of: 'NodeA', 'NodeB
ERROR: Host 'nodec' does not have an Admin Node Manager. Retry with a valid --anm_host <hostname> parameter, where where hostname is one of: 'NodeA', 'NodeB

The solution in all cases is to retry the upgrade command with a valid --anm_host parameter. For more details, see Which is the first Admin Node Manager?

API Manager port changed to HTTPS upgrading from 7.3.1

When upgrading API Gateway version 7.3.1 configuration, the API Manager traffic port (8065) is changed from HTTP to HTTPS, and the port is set to use more secure SSL settings. This means that the ciphers are set to FIPS:!SSLv3:!aNUL, and the SSLv2/v3 protocols are not allowed. The upgrade step logs the following message:

The HTTP listener for port PORT.PORTAL.TRAFFIC will be upgraded automatically to a HTTPS listener. 
If you manually reconfigure this listener to HTTP, the Try-It functionality will no longer work in API Manager.

This message is for information only and does not require any action.

Check if API Manager license required

If API Manager is installed on the old API Gateway installation you are upgrading, the upgrade step checks the new API Gateway installation for a valid API Manager license. If the license does not exist, the upgrade step logs a warning message, but continues the upgrade:

WARN: A LicenseException ('feature "apiportal" not licensed') occurred while checking for a API Manager license in 
directory '/opt/Axway-7.5.3/apigateway/conf/licenses'. 
Please contact support to acquire a new license. Place the new license in 
directory '/opt/Axway-7.5.3/apigateway/conf/licenses'.

Check if McAfee license required

If any policy in the old API Gateway installation you are upgrading contains a McAfee Anti-Virus filter, the upgrade step checks the new API Gateway installation for a valid McAfee license. If the license does not exist, the upgrade step logs a warning message, but continues the upgrade:

WARN: A LicenseException ('feature "mcafee" not licensed') occurred while checking for a McAfee License 
'/opt/Axway-7.5.3/apigateway/'. 
Please contact support to acquire a new license. Place the new license in 
directory '/opt/Axway-7.5.3/apigateway/'.

Check if RBAC permissions file has changed

Each user who logs in to the API Gateway Manager web console has a specific set of roles for Role-Based Access Control (RBAC). These determine what the user can view and what actions they can perform. The roles are defined in the acl.json file on the Admin Node Manager in the apigateway/conf/ directory.

You can manually edit this RBAC permissions file to create new roles or modify existing ones. However, the upgrade does not migrate these changes to the new installation.

The upgrade step checks for changes in the RBAC permissions file on the old API Gateway installation. If there are changes, the upgrade step logs a warning that the changes need to be manually merged into the acl.json file on the new installation:

WARN: RBAC permissions file has been modified on old system.
WARN: Differences between original file (/opt/Axway-7.5.3/apigateway/upgrade/scripts/rbac/factory/acl6.json) 
and active file (/opt/Axway-7.5.3/apigateway/upgrade/bin/out/export/rbac/conf/acl.json) 
must be manually merged into apigateway/conf/acl.json on new system.

If the RBAC changes are required to allow users to log into the Admin Node Manager or the API Gateway Manager UI, you must do this before you run the apply step, or sysupgrade will not be able to connect to the new Admin Node Manager and the apply step will fail.

Check for corrupt JARs in ext/lib

If a corrupt JAR file is located in the system-wide or instance-specific ext/lib directories in the old API Gateway installation, the upgrade step logs a warning. The upgrade does not copy the JAR to the new 7.5.3 installation.

Check for old format WSDLs

WSDL files imported to API Gateway versions earlier than 7.4.0 are flagged as having an old resource repository format. This means that they do not take advantage of the improved WSDL resource repository tracking features in later versions of API Gateway. To fix this, remove the web service and reregister the WSDL file. Broken WSDL files are discarded, but the primary WSDL URLs are retained, so you can reimport them in the Upgrade Log Analysis view in Policy Studio. For more details, see Upgrade log analysis in the API Gateway Policy Developer Guide.

Check for valid API Gateway license

API Gateway requires a valid license. If the license does not exist, or is not valid, the upgrade step logs one of the following errors:

ERROR: A LicenseException ('License not found in /home/axway/Axway-7.5.0/apigateway/conf/licenses') occurred while checking licenses in 
directory '/home/axway/Axway-7.5.3/apigateway/conf/licenses'.  
A license may not exist in this directory, or all licenses in this directory may be invalid. 
Please contact support to acquire a new license. Place the new license in 
directory '/home/axway/Axway-7.5.3/apigateway/conf/licenses'. 
Run sysupgrade again without the --clean option.
ERROR: A LicenseException ('feature "unrestricted" not licensed') occurred while checking for a server license in 
directory '/home/axway/Axway-7.5.3/apigateway/conf/licenses'. 
Please contact support to acquire a new license. Place the new license in 
directory '/home/axway/Axway-7.5.3/apigateway/conf/licenses'. 
Run sysupgrade again without the --clean option.
Note   License errors can be generated for a variety of reasons (for example, if it is tampered with, expired, for the wrong product version, or for the wrong host).

You must resolve these errors before you can continue to the apply step. If you do not place a valid license in the conf/licenses directory before proceeding to the apply step, the API Gateway will not start.

SSL certificates for management traffic regenerated

When upgrading to API Gateway 7.5.3 the SSL certificates used for management traffic between Node Managers and API Gateways are always regenerated, so that the old installation and new installation Node Managers cannot communicate with each other.

For example:

SSL certificates used between Node Managers and API Gateways for management traffic will be 
regenerated in the upgraded system. Please consult documentation if you wish to manually reconfigure the upgraded system to use the 
SSL certificates from the old version after upgrade.

This message is for information only and does not require any action. If you used custom SSL certificates in your old installation, you can manually add the certificates from your old installation to your new installation after upgrade. For more information, contact Axway Support.

Third-party JDBC JARs

If your old installation of API Gateway uses external databases for OAuth and KPS, the upgrade step logs the following warning:

WARN: OAuth/KPS tables must be upgraded in the following external 
databases: [u'jdbc:mysql://127.0.0.1:3306/testdb']. You should copy the required JDBC drivers to '/opt/Axway-7.5.1/apigateway/upgrade/lib', 
so that 'sysupgrade apply' can perform the database upgrades. Alternatively, you can upgrade OAuth/KPS tables manually using the SQL 
scripts at '/opt/Axway-7.5.1/system/conf/sql/upgrades', then run 'sysupgrade apply' with argument --skip_db_upgrade.

Metrics database reconfiguration

If you are using an external database to store API Gateway metrics in your old installation, and your old installation version is earlier than 7.4.0, the upgrade step logs the following warning.

WARN: Metrics are being written from API Gateway to the following 
database: [u'jdbc:mysql://127.0.0.1:3306/testdb']. From 7.4.0 onwards, metrics are written from the Node Manager, not the API Gateway. You will need 
to update the database schema and reconfigure metrics after upgrade. 

This indicates that you need to reconfigure metrics after the upgrade, because in API Gateway 7.4.0 and later versions, the Node Manager writes metrics instead of the API Gateway. For more information on using dbsetup to upgrade the metrics database, see Upgrade API Gateway Analytics.

ActiveMQ directory

If the old installation of API Gateway uses embedded ActiveMQ, the upgrade step might log some of the following warnings.

ActiveMQ directory not set

If the directory for ActiveMQ data is not set in the old installation, the following warning appears:

WARN: The ActiveMQ directory is not configured correctly, it is set to '', sysupgrade will NOT export the data.

In this case, sysupgrade proceeds with the upgrade. You should fix the ActiveMQ configuration in the old installation.

ActiveMQ directory does not exist

If the directory for ActiveMQ data set in the old installation does not exist, warnings similar to the following might appear.

WARN: The absolute ActiveMQ directory '/home/user1/does-not-exist' does not exist, sysupgrade will NOT export the data.
WARN: The ActiveMQ directory '/home/user1/AxwayInstalls/GOLD/7.3.1/apigateway/groups/group-2/instance-1/does-not-exist' 
does not exist, sysupgrade will NOT export the data.

In both of these cases, sysupgrade proceeds with the upgrade. You should fix the ActiveMQ configuration in the old installation.

ActiveMQ directory not under API Gateway installation directory

If the directory set for ActiveMQ data in the old API Gateway installation is not a directory under the API Gateway installation path, the following warning appears:

WARN: The absolute ActiveMQ directory '/home/user1/activemq/data' is NOT a directory under the old version install 
directory '/home/user1/AxwayInstalls/GOLD/7.4.1/apigateway'. The new version system will use the same directory. When the 
new version API Gateway uses the data in this directory it may become unusable by the old system. You should take a manual 
backup of the contents of this directory before proceeding in case you need to run the old version API Gateway again. Sysupgrade will keep a backup of 
the ActiveMQ data in '/home/user1/dev/sysupgrade1_feature/vordel/install/staging/images/apigateway/upgrade/bin/out/export/activemq/backup' 
but this may be removed as a result of a clean command, or export --force.

In this case, the upgrade proceeds, and the new API Gateway installation uses the same directory for ActiveMQ data after upgrade.

The new installation might change the data in this directory so that it is no longer usable by the old installation. The sysupgrade process makes a backup of the data in the upgrade/bin/out/export/activemq/backup directory. However, this backup might be removed if you use the clean or export --force commands. We recommend that you create an additional backup to another location in case you need to revert back to using the old installation at any stage.

Check for duplicate API Manager data

If API Manager is running in the old installation, exported KPS data is checked for duplicate users called apiadmin and duplicate organizations called Community. If duplicate data is found, this indicates that KPS is not configured correctly in the old installation.

The upgrade step logs the following error message:

[ERROR] messages from [API Manager] :
ERROR: Found duplicate 'apiadmin' users in KPS table: 
/opt/Axway-7.5.3/apigateway/upgrade/bin/out/upgrade/kps/groups/group-2/instance-1/conf/kps/backup/sysexport_api_portal_portaluserstoreldap_json
ERROR: Found duplicate 'Community' organizations in KPS table: 
/opt/Axway-7.5.3/apigateway/upgrade/bin/out/upgrade/kps/groups/group-2/instance-1/conf/kps/backup/sysexport_api_portal_portalorganizationstoreldap_json
ERROR: There is a KPS configuration problem on the old system. This must be resolved 
before upgrade can proceed. Please contact Axway support for assistance.

To resolve this issue, contact Axway Support.

Check for valid FIPS license

If the old API Gateway installation is FIPS-enabled, a FIPS-enabled license is required for the new installation. If the license does not exist, or is not valid, the upgrade step logs the following error:

ERROR: A LicenseException ('feature "FIPS" not licensed') occurred while checking for a FIPS license in directory '/home/user1/dev/apigateway/conf/licenses'. 
Please contact support to acquire a new license. Place the new license in directory '/home/user1/dev/apigateway/conf/licenses'.

You must resolve this issue before you can continue to the apply step.

Check for Apache Cassandra-backed collections

If you run the sysupgrade upgrade command with the --no_cassandra option, the configuration is checked for Cassandra-backed KPS collections.

If Cassandra-backed collections are found, the upgrade step logs the following error:

ERROR: Group 'Default Group' contains KPS collections with Cassandra data source: ['My Collection']. 
Either run sysupgrade without '--no_cassandra' option, or remove Cassandra dependency from 
source system by deleting collections or moving them to a non-Cassandra data source.

You must resolve this error before you can proceed to the apply step.

To resolve this error, delete or update the Cassandra-backed KPS collections in your old installation as detailed in Delete or update Apache Cassandra-backed KPS collections in the old installation, and then rerun the export and upgrade steps.

Alternatively, you can rerun the upgrade step without the --no_cassandra option. However, your new installation will require an Apache Cassandra database cluster as detailed in Configure an Apache Cassandra database cluster in the new installation.

Apply command errors and warnings

This section describes example errors and warnings that the sysupgrade apply command generates when applying the upgraded data to the new API Gateway installation.

Errors and warnings are logged to the following file:

Axway-7.5.3/apigateway/apply/bin/out/logs/apply.log

Incorrect Admin Node Manager host

If the --anm_host specified is not the first Admin Node Manager host and the first Admin Node Manager host is not running, the apply step logs the following error:

ERROR: Node Manager error: Failed to sign CSR for service. None of the available Admin Node Managers have the domain private key, 
or were able to unlock it with provided passphrase.

The solution is to always use the same --anm_host value for the first Admin Node Manager. For more details, see API Gateways missing from topology after upgrade.

Related Links