• Application Layer
• Virtual Machine
Database Layer
Begin all investigations by looking at the multi-application table at the top of the SOC dashboard. Here you can see an overview of the service or tier, including name, service level compliance, and user health status.
Only one service is shown in this example, but in a typical environment, there would be several.
Tip You can add and remove columns to customize the information that appears in this table by clicking the customizer icon to the right of the Search box.
Chapter 5—Using the SOC for APM Triage
To triage an application:
1 Review the Service Level Compliance and User columns of the table to locate the tier of the service that is experiencing performance degradation.
In this case, the User tier icon indicates an issue for users.
The service level compliance icon indicates that the service has not yet violated the service level compliance policy in Foglight. However, if the issue persists, this icon may change to indicate a violation.
2 Review the End User tab to determine the impact on users.
End user tiles are grouped by transaction: MD1Patient (top row), MD1Physician (middle row), and MD1Admin (bottom row). They are also ordered by severity (that is, the users with the worst health status or most alarms appear at the top).
Here you see that one transaction is having an issue, with both the real user tile and synthetic tiles reflecting an issue when accessing MD1Patient. The users accessing MD1Physician and
MD1Admin are not experiencing any issues (their status is green across the board), which
indicates that the issue is not affecting the whole application, only the MD1Patient transaction.
The Trace Analysis health status ( ) icon on the MD1Patient Real User tile indicates that there is a problem somewhere in the trace analysis metrics.
Foglight has also generated critical level alarms ( ) for all four MD1Patient transactions. Since all synthetic transactions are affected, this is not a location-specific issue. Synthetic transactions can be used as a benchmark for healthy performance. In this case, the fact that the synthetics are affected indicates that this is not a problem caused by a one-time error from a single user; it is a recurring or ongoing problem affecting all users.
Monitoring Application Performance: User and Reference Guide 40 Chapter 5—Using the SOC for APM Triage
3 Click the title bar of the affected Real Users tile to drill down for more information. The Real User Performance detail view opens.
This detail view captures several key metrics in chart format. Consider the following:
Metric Key Insight
Response Time Beginning at 09:35, users are experiencing a longer response time. Page Requests At 09:35, the number of page requests is slightly lower than earlier
peak loads. This indicates that the problem is not due to a spike in user activity.
SLA/OLA
Attainment Also at Level Agreement (OLA) are still being met, but the percentage is 09:35, the Service Level Agreement (SLA) and Operating
decreasing.
Front End /
Back End The time the transaction spent in the back end (that is, in the supporting architecture) has increased, and the problem appears to
Chapter 5—Using the SOC for APM Triage
Note Click the title bar of the MD1Patient Hits Over 7 Seconds chart to drill down into the individual sessions recorded by FxV. The FxV details allow you to see the Hit URLs and error messages, and to drill down into individual sessions, where you can step through a session to locate the error point.
In this case, drilling down to the sessions shows that this is a content error issue. Some users are seeing error messages in their web browsers. This increases the priority of this issue.
This looks like a serious problem that is impacting many users, and the issue seems to be originating in the back end.
Close this view by clicking the ‘x’ in the upper right corner.
4 Next, investigate the application topology to determine where in the supporting architecture the problem originates.
Click the Dependencies tab to view the application topology.
Tip Always work from left to right when triaging dependencies.
Overall, the infrastructure is in good health — all of the hosts have a green check mark, indicating that their status is normal. However, several of the tiers that contain platform and code components appear to have issues that must be investigated.
Trace Analysis The overall number of hits has decreased, as shown in the MD1Patient Hits chart.
The number of users experiencing delay of more than seven seconds in their transaction execution has increased, as shown in the
MD1Patient Hits Over 7 Seconds chart.
Monitoring Application Performance: User and Reference Guide 42 Chapter 5—Using the SOC for APM Triage
The Web tier has no issues. Moving to the right, you see that there is a warning on the Application (MedRecApp) tier.
Tip Hover the mouse pointer over a host icon to open a popup view of the application components.
5 Click the application tier title bar to drill down for more details.
The MedRecApp Details view opens. The banner section of the Summary tab displays the service level compliance for the tier. The lower portion of the view contains a tile for each application component, grouped by host.
Tip For more information about the metrics displayed on views and tiles, see “Reference” on page 67.
Chapter 5—Using the SOC for APM Triage
On the Request Types tab, you see that POST/medrec/patient/viewPatient.action is the
source of the warning. Select this request.
Tip Use the Search box to filter the list of requests. Type medrec/patient to include only those requests with that string in the name.
The response time for this request has increased and remains high. The Bottleneck Tier Name column lists the database tier as the source of the bottleneck.
Note There are additional columns that may be of interest that are hidden by default. To show them, click the customizer icon to the right of the Search box and select the columns you want to display.
The Execution Time chart at the bottom of the Application Details view (below the list of request types) indicates that the most time is spent in the database tier. It also shows that the time spent in the database tier began to increase at 09:30, which corresponds to the increase
in the time spent in the back end that appeared in the Real User Performance view in step 3. This evidence all points to a problem in the database tier, which is where you should look next.
Click the ‘x’ in the upper-right corner to close the Application Details view. Close the
MedRecApp Details view as well.
7 On the Dependencies tab, click the title bar of the MedRecDB database tier to drill down for more details. The details view for the database tier opens.
Monitoring Application Performance: User and Reference Guide 44 Chapter 5—Using the SOC for APM Triage
There is a critical alarm on the application component tile for the Oracle database. The Oracle Tile displays the top four session bottlenecks as color-coded bars. The larger the main color bar, the higher the percentage of resources were spent on that metric. The thin blue lines on the top of each bar indicate the range for the normal state. The thin green/ yellow/orange bar below the main color bar indicate the metric’s trend compared against itself.
In this case, the database has Lock wait issues.
Now that you have determined the most probable source, you can engage the database administrator to evaluate the problem.
In this scenario, the database and application administrators should have been aware of the problems before the Application Performance Manager became involved. If the administrators were using Foglight, they would have received alarms generated and sent by Foglight.
Chapter 5—Using the SOC for APM Triage