Service Mesh Traceability

Observe transactions in mesh.

7 minute read

Before you begin

Before you start, see Deploy your agents with the Axway CLI to learn how to use the CLI to install the Istio agents into your Kubernetes cluster.

This page will reference the resources created from the Deploy your agents with the Axway CLI procedure.

Prerequisites

These prerequisites are required by the Axway Central CLI, which you will use to configure the Amplify Istio Discovery Agent.

Node.js >= 10.13.0 and <= 12.14.1
Minimum Axway Central CLI version: 1.7.0 or later

For more information, see Install Axway Central CLI.

Overview

The Amplify Istio Traceability Agent is installed into your Kubernetes cluster as part of deploying the ampc-hybrid helm chart. The Traceability Agent (TA) sends metrics and logs for API activity back to Amplify Central so that you can monitor service activity and troubleshoot your services. The agent publishes a summary of the transaction which can be seen in Business Insights. Once the transaction summary is expanded, you can see all the related spans within a transaction including the request and response headers for each.

The Amplify Istio Traceability Agent has two modes; default and verbose. The default mode captures only the headers specified in the EnvoyFilter and the verbose mode captures all the headers in request and response flows.

Setup

The Amplify Istio Traceability Agent logs and publishes traffic within the Mesh. In order to generate traffic, we need to create certain custom resource definitions (CRDs) in the mesh.

Amplify Central resources

In order to better filter transactions related to the services in the mesh, certain resources need to be created for each service running on the mesh - namely APIService, APIServiceRevision and APIServiceInstance. The APIService needs to include the attribute “externalAPIID” in its definition.

The format of externalAPIID is:

clustername-httproutename

The “externalAPIID” attribute is used to define the correlation between the Kubernetes cluster name and the http route name on the VirtualService (that is created in the next section) so that the traffic for this APIService can be easily filtered out.

The cluster name is the value of the field als.clusterName in the agent override file hybrid-override.yaml generated by the Axway CLI during Istio agent installation to your Kubernetes cluster. Choose a name for the http route name - ideally the same as the name for APIService or very similar to it. Make a note of this route name, as you will need it in the next section.

Next, create the following resources in Central using the Axway CLI:

Note

Create these resources manually only if you are running standalone Amplify Istio Traceability Agent. If you are running Amplify Istio Traceability Agent along with the Discovery agents, these resources are automatically created and you can skip to Istio CRDs.

kind: APIService
name: <<serviceName>>
metadata:
  scope:
    kind: Environment
    name: <<environmentName>>
attributes:
  externalAPIID: <<clustername-http.name>> # http.name to be used for VS
spec: {}
---
kind: APIServiceRevision
name: <<revisionName>>
metadata:
  scope:
    kind: Environment
    name: <<environmentName>>
spec:
  apiService: <<serviceName>>
  definition:
    type: "oas2"
    value: <<base64 encoded spec>>
---
kind: APIServiceInstance
name: <<serviceName>>
metadata:
  scope:
    kind: Environment
    name: <<environmentName>>
spec:
  apiServiceRevision: <<revisionName>>
  endpoint:
    - host: "apicentral.axway.com"
      port: 8080
      protocol: http
      routing:
        basePath: <<basePath>>

Here is an example for the Hybrid List demo service:

kind: APIService
name: mylist
metadata:
  scope:
    kind: Environment
    name: meshone
attributes:
  externalAPIID: clusterone-mylist
spec: {}
---
kind: APIServiceRevision
name: list-v1
metadata:
  scope:
    kind: Environment
    name: meshone
spec:
  apiService: mylist
  definition:
    type: "oas2"
    value: eyJzd2FnZ2VyIjoiMi4wIiwiaW5mbyI6eyJ0aXRsZSI6Im15bGlzdCIsImRlc2NyaXB0aW9uIjoiQW4gQVBJIEJ1aWxkZXIgc2VydmljZSIsInZlcnNpb24iOiIxLjAuMCJ9LCJob3N0IjoiMTAwLjk5LjIyMi4xMDI6ODA4MCIsImJhc2VQYXRoIjoiL2FwaSIsInBhdGhzIjp7Ii9saXN0Ijp7ImdldCI6eyJyZXNwb25zZXMiOnsiMjAwIjp7ImRlc2NyaXB0aW9uIjoiT0siLCJzY2hlbWEiOnsidHlwZSI6ImFycmF5IiwiaXRlbXMiOnsiJHJlZiI6IiMvZGVmaW5pdGlvbnMvbGlzdCJ9fX19fSwicG9zdCI6eyJyZXNwb25zZXMiOnsiMjAwIjp7ImRlc2NyaXB0aW9uIjoiT0siLCJzY2hlbWEiOnsidHlwZSI6Im9iamVjdCIsInByb3BlcnRpZXMiOnt9fX19LCJwYXJhbWV0ZXJzIjpbeyJpbiI6ImJvZHkiLCJuYW1lIjoiYm9keSIsInNjaGVtYSI6eyIkcmVmIjoiIy9kZWZpbml0aW9ucy9saXN0In19XX19LCIvbGlzdC97aWR9Ijp7ImdldCI6eyJyZXNwb25zZXMiOnsiMjAwIjp7ImRlc2NyaXB0aW9uIjoiT0siLCJzY2hlbWEiOnsiJHJlZiI6IiMvZGVmaW5pdGlvbnMvbGlzdCJ9fX0sInBhcmFtZXRlcnMiOlt7ImluIjoicGF0aCIsIm5hbWUiOiJpZCIsInR5cGUiOiJzdHJpbmciLCJyZXF1aXJlZCI6dHJ1ZX1dfSwicGFyYW1ldGVycyI6W3sibmFtZSI6ImlkIiwiaW4iOiJwYXRoIiwidHlwZSI6InN0cmluZyIsInJlcXVpcmVkIjp0cnVlfV0sImRlbGV0ZSI6eyJyZXNwb25zZXMiOnsiMjAwIjp7ImRlc2NyaXB0aW9uIjoiT0siLCJzY2hlbWEiOnsidHlwZSI6Im9iamVjdCIsInByb3BlcnRpZXMiOnt9fX19LCJwYXJhbWV0ZXJzIjpbeyJpbiI6InBhdGgiLCJuYW1lIjoiaWQiLCJ0eXBlIjoic3RyaW5nIiwicmVxdWlyZWQiOnRydWV9XX19fSwiZGVmaW5pdGlvbnMiOnsibGlzdCI6eyJ0eXBlIjoib2JqZWN0IiwidGl0bGUiOiJsaXN0IiwicHJvcGVydGllcyI6eyJuYW1lIjp7InR5cGUiOiJzdHJpbmcifSwicHJpY2UiOnsidHlwZSI6InN0cmluZyJ9LCJzdG9yZSI6eyJ0eXBlIjoic3RyaW5nIn19fSwiUmVzcG9uc2VNb2RlbCI6eyJ0eXBlIjoib2JqZWN0IiwicmVxdWlyZWQiOlsic3VjY2VzcyIsInJlcXVlc3QtaWQiXSwiYWRkaXRpb25hbFByb3BlcnRpZXMiOmZhbHNlLCJwcm9wZXJ0aWVzIjp7ImNvZGUiOnsidHlwZSI6ImludGVnZXIiLCJmb3JtYXQiOiJpbnQzMiJ9LCJzdWNjZXNzIjp7InR5cGUiOiJib29sZWFuIiwiZGVmYXVsdCI6ZmFsc2V9LCJyZXF1ZXN0LWlkIjp7InR5cGUiOiJzdHJpbmcifSwibWVzc2FnZSI6eyJ0eXBlIjoic3RyaW5nIn0sInVybCI6eyJ0eXBlIjoic3RyaW5nIn19fSwiRXJyb3JNb2RlbCI6eyJ0eXBlIjoib2JqZWN0IiwicmVxdWlyZWQiOlsibWVzc2FnZSIsImNvZGUiLCJzdWNjZXNzIiwicmVxdWVzdC1pZCJdLCJwcm9wZXJ0aWVzIjp7ImNvZGUiOnsidHlwZSI6ImludGVnZXIiLCJmb3JtYXQiOiJpbnQzMiJ9LCJzdWNjZXNzIjp7InR5cGUiOiJib29sZWFuIiwiZGVmYXVsdCI6ZmFsc2V9LCJyZXF1ZXN0LWlkIjp7InR5cGUiOiJzdHJpbmcifX19fX0=
---
kind: APIServiceInstance
name: mylist
metadata:
  scope:
    kind: Environment
    name: mesh
spec:
  apiServiceRevision: list-v1
  endpoint:
    - host: "apicentral.axway.com"
      port: 8080
      protocol: http
      routing:
        basePath: "/mylist"

Once configured, use the following command to populate the resources in Amplify Central:

axway central apply -f <fileName>.yaml

Istio CRDs

Gateway

First, create a Gateway in the namespace in which the Istio agents were installed. If you already have a Gateway CRD, you can skip to Virtual Service and specify that Gateway in the Virtual Service.

In the example below, the selector is specified as the “istio-apic-ingress” (the Ingress gateway that is installed during the Istio installation step in Deploy your agents with the Axway CLI). If you have a separate Ingress gateway that you would like to use, change the spec.selector.istio field to the label of that Ingress gateway instead.

Note

For more information about Gateway CRD, please refer to Istio documentation.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway-ingress
  namespace: <<apic-control>>
spec:
  selector:
    istio: istio-apic-ingress
  servers:
  - hosts:
    - <<cluster-name>>.hybrid.sandbox.axwaytest.net
    port:
      name: <<port-HTTP>>
      number: <<8080>>
      protocol: <<HTTP>>

For an HTTPS Gateway, use the following configuration:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: gateway-ingress
  namespace: <<apic-control>>
spec:
  selector:
    istio: istio-apic-ingress
  servers:
  - hosts:
    - <<cluster-name>>.hybrid.sandbox.axwaytest.net
    port:
      name: port-HTTP
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE 
      serverCertificate: /etc/istio/istio-ingressgateway/tls.crt
      privateKey: /etc/istio/istio-ingressgateway/tls.key

Once configured, create the resource using the command:

 kubectl apply -f <fileName>.yaml

Virtual Service

Next, create the Virtual Service for the included demo service within the mesh. Unless you have already taken note of the ‘http route name’ part of the externalAPIId attribute associated with the APIService from the previous section, the value for it needs to be extracted from the APIService.

First, locate the APIService and make a note of its name and the name of the environment under which it is scoped:

axway central get apisvc

Then, using this command, print the value for externalAPIID and make a note of the ‘http route name’ part of it:

axway central get apisvc <APIService_name> -s <Environment_name> -o yaml | grep externalAPIID

Example:

axway central get apisvc
✔ Resource(s) successfully retrieved

NAME    AGE         TITLE   RESOURCE KIND  SCOPE KIND   SCOPE NAME          RESOURCE GROUP
mylist  2 days ago  mylist  APIService     Environment  cli-1618415896316   management

axway central get apisvc mylist -s cli-1618415896316 -o yaml | grep externalAPIID
  externalAPIID: mycluster-mylist

The following example, used to create a VirtualService, applies to the “list” demo service that comes with the Istio agent Helm installation. If you already have a Virtual Service, you can skip to the Pre-existing Virtual Services section below:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: route-to-list
  namespace: apic-control
spec:
  hosts:
  - "*"
  gateways:
  - gateway-ingress
  http:
  - name: mylist # this should be gotten from the http route name part of the externalAPIId on APIService
    match:
    - uri:
        prefix: /mylist
    rewrite:
      uri: /api
    route:
    - destination:
        host: ampc-hybrid-list.apic-demo.svc.cluster.local
        port:
          number: 8080

Once configured, create the resource using the command:

 kubectl apply -f <fileName>.yaml

Pre-existing Virtual Service

If you have a Virtual Service resource already, simply add a name for (or rename) the http route so that the API Service and the related transactions can be linked in API Central:

Example:

  http:
  - name: mylist

Note

The name specified under the http.name field of the VirtualService should be the same as the ‘http route name’ part of the externalAPIId attribute on the APIService.

Service Entry

If you have an egress hop from a service in the mesh, then create a service entry. See the example below:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin.org
  namespace: apic-control
spec:
  hosts:
  - httpbin.org
  ports:
  - name: http-80
    number: 80
    protocol: HTTP
  resolution: DNS

The setup is complete for observability in the mesh. To view transactions in Business Insights, generate some traffic for the Hybrid List demo service:

curl -v http://demo.sandbox.axwaytest.net:8080/mylist/list

Amplify Central control plane

Toggling the Traceability Agent

After deploying the ampc-hybrid helm chart to your Kubernetes cluster, you can see the Amplify Istio Traceability Agent running. The service is called ampc-hybrid-als. During the step Deploy your agents with the Axway CLI, you were able to select the mode for the Amplify Istio Traceability Agent. If you want to switch the mode, use the following procedure.

From default to verbose:

Edit the istio-override.yaml file’s configuration under the meshConfig section to set enableEnvoyAccessLogService as true:

spec:
  meshConfig:
    enableTracing: true
    enableEnvoyAccessLogService: true

After the change, re-install Istio again:

 istioctl install --set profile=demo -f istio-override.yaml

After the Istio re-installation, run the following command to set the Amplify Istio Traceability Agent’s mode to “verbose”:

helm repo update
helm upgrade --install --namespace apic-control ampchybrid axway/ampc-hybrid -f hybrid-override.yaml --set als.mode="verbose"

From verbose to default:

Edit the Istio-override.yaml file’s configuration under the meshConfig section to set enableEnvoyAccessLogService as false:

spec:
  meshConfig:
    enableTracing: true
    enableEnvoyAccessLogService: false

After the change, re-install Istio again:

 istioctl install --set profile=demo -f istio-override.yaml

After the Istio re-installation, run the following command to set the Amplify Istio Traceability Agent’s mode to “default”:

helm repo update
helm upgrade --install --namespace apic-control ampc-hybrid axway/ampc-hybrid -f hybrid-override.yaml --set als.mode="default"

In default mode the Traceability Agent can be configured to only capture certain request and response headers. By default, we capture all the headers specified in the EnvoyFilter configuration below. See “additional_request_headers_to_log” and “additional_response_headers_to_log” section.

 apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: patch-gateway-and-sidecars-with-als
  namespace: <<envoyFilterNamespace>>
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: MERGE
        value:
          name: "envoy.filters.network.http_connection_manager"
          typed_config:
            "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager"
            access_log:
              - name: envoy.access_loggers.http_grpc
                typed_config:
                  "@type": "type.googleapis.com/envoy.extensions.access_loggers.grpc.v3.HttpGrpcAccessLogConfig"
                  additional_request_headers_to_log: ["accept","user-agent","x-envoy-decorator-operation","x-envoy-external-address","x-forwarded-client-cert","x-forwarded-for","x-forwarded-proto","x-request-id","x-b3-parentspanid","x-b3-spanid","x-istio-attributes"]
                  additional_response_headers_to_log: ["connection","content-length","content-md5","content-type","date","etag","date","request-id","response-time","server","start-time","vary"]
                  common_config:
                    filter_state_objects_to_log: ["wasm.upstream_peer","wasm.upstream_peer_id","wasm.downstream_peer","wasm.downstream_peer_id"]
                    log_name: mesh
                    grpc_service:
                      google_grpc:
                        target_uri: ampc-hybrid-als.apic-control.svc.cluster.local:9000
                        stat_prefix: ampc-hybrid-als

To exclude any headers, remove them from “additional_request_headers_to_log” and “additional_response_headers_to_log”. Please note that unless otherwise specified envoyFilterNamespace is “istio-system”. Once the configuration is changed, run the following command:

 kubectl apply -f <fileName>.yaml

Transaction Redaction

The Traceability Agent enforces redaction by default. The agent can be configured to show certain paths, query parameters, and header information based on redaction environment variables provided to it. For instructions on how to set the redaction configuration, see Trace Redaction.

Once the environment variables are set, put them in a helm override configuration:

als:
  redaction:
    path:
      show: ${TRACEABILITY_REDACTION_PATH_SHOW:[]}
    queryArgument:
      show: ${TRACEABILITY_REDACTION_QUERYARGUMENT_SHOW:[]}
      sanitize: ${TRACEABILITY_REDACTION_QUERYARGUMENT_SANITIZE:[]}
    requestHeader:
      show: ${TRACEABILITY_REDACTION_REQUESTHEADER_SHOW:[]}
      sanitize: ${TRACEABILITY_REDACTION_REQUESTHEADER_SANITIZE:[]}
    responseHeader:
      show: ${TRACEABILITY_REDACTION_RESPONSEHEADER_SHOW:[]}
      sanitize: ${TRACEABILITY_REDACTION_RESPONSEHEADER_SANITIZE:[]}

Below is a sample redaction configuration:

als:
  redaction:
    path:
      show: '[{keyMatch:".*"}]'
    requestHeader:
      show: '[{keyMatch:".*"}]'

The configuration above will display all URI path information and all request headers.

Put your redaction configuration into a file and then execute the following command:

helm upgrade --install ampc-hybrid axway/ampc-hybrid --namespace apic-control -f hybrid-override.yaml -f <pathToConfigFile>/config.yaml

Monitor whether the Amplify Istio Traceability Agent pods have restarted by executing the following command:

kubectl -n <namespace of Amplify Istio Traceability Agent> get pods

The deployment of Amplify Istio Traceability Agent will fail if invalid configuration is provided. If there is an error in the pods after executing the command above, you can check the log by executing the following command:

kubectl -n <namespace of Amplify Istio Traceability Agent> logs <podName>

The logs should display the configuration error. Fix the configuration and repeat the steps above.

Last modified March 9, 2022: VIZ-61 rename api observer to business insights (#90) (420cd03)

Product

Content Type

Environment

Amplify Central

Service Mesh Traceability

Before you begin

Prerequisites

Overview

Setup

Amplify Central resources

Note

Istio CRDs

Gateway

Note

Virtual Service

Pre-existing Virtual Service

Note

Service Entry

Toggling the Traceability Agent

Transaction Redaction

Related Links