Analytics empty data

[jsanhc]

Description

The analytics UI doesn't show any data about usage endpoints and psu counter because the below entries in mongodb are empty:

• endpointUsageEntry
• psuCounterEntry

Other UI components don't show data in the analytic APP.

Task

• Investigate empty entries.
• Investigate UI components don't show data in the app.
• Apply a fix to store the entries data
• Prepare upgrade step class for the next deployment version.

[jsanhc]

Investigation: Investigate empty entries.

Results

psuCounterEntry

This entry was empty because DataApiController used by the service rs_ui the @autowire object PsuCounterEntryKPIService was injecting the default implementation NoOpPsuCounterEntryKPIService (analytics-client dependency) where the implementation is empty and therefore the entry is never saved.

Suggested solution

• The default implementation of PsuCounterEntryKPIService came from the dependency analytics-client added from the parent, the solution must be add the analytics-webclient dependency to openbanking-reference-implementation/forgerock-openbanking-aspsp/forgerock-openbanking-aspsp-rs/forgerock-openbanking-rs-ui to inject the proper implementation of PsuCounterEntryKPIService, the @service object WebClientPsuCounterEntryKPIService.

<dependency>
      <groupId>com.forgerock.openbanking.clients</groupId>
      <artifactId>forgerock-openbanking-analytics-webclient</artifactId>
</dependency>

endpoitUsageEntry

UPDATE: Never will be use for the next releases (date: 2021-03-09)

Use the below investigation only such as information, no solution will be applied.

The current version of the rest api controller EndpointUsageKpiAPIController from analytics core module from analytics used to save the entries sent by any service (as-api, rs-ui, etc etc) using the Filter MetricFilter from clients doesn't use any more the repository EndpointUsageEntriesRepository to save the object entry EndpointUsageEntry.
The @component EndpointUsageAggregator use the MongoTemplate upsert operation to update the endpointUsageAggregate entries only.

/* Before ob-reference-implementation (forgecloud) commit 603a0c0e */
/* date 10/06/2019 */
/* Class com.forgerock.openbanking.metrics.api.endpoint.EndpointUsageKpiAPIController */
private void saveEntries(List<EndpointUsageEntry> endpointUsageEntries) {
        endpointUsageEntries.forEach(endpointUsageEntryEnricher::enrich);
        endpointUsageEntriesRepository.saveAll(endpointUsageEntries);
        endpointUsageAggregator.aggregate(endpointUsageEntries);
}
/* Current version on open banking tool kit */ 
public void saveEntries(List<EndpointUsageEntry> endpointUsageEntries) {
        endpointUsageEntries.forEach(endpointUsageEntryEnricher::enrich);
        endpointUsageAggregator.aggregate(endpointUsageEntries);
}

It means that currently the endpointUsageEntry will be empty in mongodb always because never is saved any entry.
This changes seems to be related with performance improvements.

Suggested Solutions

1. Update analytics to consume the data from endpointUsageAggregate.
2. Hide the UI section where must show the endpointUsageEntry data.
3. Reimplement the use of repository EndpointUsageEntriesRepository to save again the endpointUsageEntry and avoid the empty data. Beware: because the use of EndpointUsageEntriesRepository.save it has been deleted from the controller for performance reasons
   endpoitUsageEntry UPDATE

Not used any more the entry endpointUsageEntry

[jsanhc]

Investigation: Investigate UI components don't show data in the app.

Results

TAB UI: General Service Usage

Components:

• Number of endpoint calls by Open Banking API versions
• Number of endpoint calls by Open Banking API types

The UI consume the metric service to get the data and to do that each request contains sort, filters and fields and the service make a data filter and build an aggregation data using these values.

• Sometimes the Request contains values that don't exist and the aggregation data it's null.
  • Example: Filter by ob version, the endpointUsageAggregate doesn't have that field but must have it, the problem it's related with the annotation class OpenBankingAPI used in MetricFilter (clients), this class doesn't match with the origin class used to annotate the controllers in aspsp, it's duplicated using different packages in different projects.

    OpenBankingAPI annotation = AnnotationUtils.findAnnotation(handler.getBeanType(), OpenBankingAPI.class);

And the below source never works to find the values on the annotation class, superclass or interfaces.

    @Override
    @SuppressWarnings("unchecked")
    public <A extends Annotation> A getDeclaredAnnotation(Class<A> annotationClass) {
        Objects.requireNonNull(annotationClass);

        return (A) annotationData().declaredAnnotations.get(annotationClass);
    }

Solution

• Create a new module to centralise the annotations on openbanking-commons named forgerock-openbanking-annotations and move all openbanking custom annotation OpenBankingAPI to the new module.
• Make a refactor on all classes to replace and to use the new annotations library from openbanking-commons through all repositories that use the annotations.

TAB UI: TPP

Components:

• APIs performances per TPP

The initial Filter sent in the request:

  initialFilters: ITableFilterList = [
    {
      field: 'userType',
      regex: 'OIDC_CLIENT'
    },
    {
      field: 'tppEntry',
      notNull: true
    }
  ];

The field tppEntry doesn't exist and the value of userType always is ANONYMOUS unless the header UserType is sent with another value like as OIDC_CLIENT.

Possible solution

• Modify the query to fix the aggregate group on analytics-core

TAB UI: Security

Components:

• Number of requests to the /authorise endpoint

The initial Filter sent in the request:

  initialFilters: ITableFilterList = [
    {
      field: 'application',
      regex: 'as-api'
    }
  ];

The field application doesn't exist to apply that filter.

[jsanhc]

Investigation: Investigate UI components don't show data in the app.

TAB UI: Read/Write APIs menu / Payments APIs

Components:

• Payments endpoints called by OIDC clients

The query return empty data because the entry consentStatusEntry doesn't have any document that match with the filter:

{ "consentType": "PAYMENT_FILE_CONSENT" }

Tasks

• Investigate why the consent file payment is not being saved in the analytics entry such as the others.
• Fix to save the payment file consent in the analytics entry properly.

Investigation

The problem was that the collection of postman used to test the functionalities stoped with error before run the File payment test for an error on international standing orders test and for that reason the File payment test doesn't run and doesn't create the metric entry for analytics.

Fix

• Fixing the collection on postman all works properly.

[jsanhc]

Investigation: Investigate UI components don't show data in the app.

TAB UI: Read/Write APIs menu / Confirmation of funds APIs

Components:

• ALL UI components empty

Task

• Investigate all UI components

Investigation

The problem was that the collection of postman used to test the functionalities stoped with error before run the Confirmation of funds test for an error on international standing orders test and for that reason the Confirmation of funds test doesn't run and doesn't create the entry metric entry for analytics.

Fix

• Fixing the collection on postman all works properly.

TAB UI: Read/Write APIs menu / Events notification APIs

Components:

• ALL UI components empty

Task

• Investigate all UI components

Investigation

The problem was that the collection of postman used to test the functionalities stoped before run the Notification test for an error on international standing orders test and for that reason the Notification test doesn't run and doesn't create the entry metric entry for analytics.

Fix

• Fixing the collection on postman all works properly.