Obtaining data from DMS in an automated fashion, even with the WLST support, can be a challenge. In this blog I provide a Python 2.7 script which allows you to get information from the DMS and dump it in a CSV file for further processing. The script first logs and uses the obtained session information to download information from a specific table in XML. This XML is converted to CSV. The code does not require an Oracle Home (it is not WLST based). The purpose here is to provide an easy to use starting point which can be expanded to suit specific use-cases. The script works against WebLogic 11g and 12c environments (has been tested against 11.1.1.7 and 12.2.1). Do mind that the example URL given in the script obtains performance data on webservice operations. This works great on composites but not on Service Bus or JAX-WS services. You can download a general script here (which requires minimal changes to use) and a (more specific) script with examples of how to preprocess data in the script here.
How to work with the DMS
The dynamic contents of the DMS tables (like average service response times) are reset upon server restart. Static contents such as deployed composites, remain comparable even after a restart. The contents can also be reset by a script. See for example here. An easy way to work with the DMS is to first reset it, perform some tests and then collect data. After collecting data, you can again reset it and start with a next test.
The output of the script can be piped to a (CSV) file and then opened in Excel. There you can make a graph of the data to for example analyse the poorest performing operations or look at JVM garbage collects, datasource statistics, etc. You can also easily expand the script to do measures over time or put the output in a database table (see an example here on how to do JDBC from Jython/WLST).
Requirements for running the script
The Python script requires the 'requests' module to be installed. This module takes care of maintaining the DMS session. Installing this module can be challenge if the system you want to run the script from, has limited privileges or connectivity. Using WinPython might help. This distribution of Python is portable and can be copied standalone to a Windows folder or share. It will work just fine from there and is not dependant on Windows registry settings or specific OS configuration. I recommend you first prepare a WinPython folder with the requests module already installed and package that with your script if you have such a requirement.
Installing the module can be done with (supplied with the WinPython distribution (and almost every other Python distribution) in the Scripts folder) 'pip install requests' or 'easy_install requests'. If your customer is using a proxy server (which does not require NTLM authentication, otherwise consider NTLMaps, cNTLM or talk operations into using a non-proprietary authentication protocol), you can set the following environment variables:
set http_proxy=http://username:password@<host>:<port>
set https_proxy=https://username:password@<host>:<port>
After you have installed the module and the environment you want to connect to from the script, does not require you to go through the proxy server, do not forget to unset these.
set http_proxy=
set https_proxy=
Inside the script you should specify host, port, username and password of the WebLogic environment. The user should have administrative privileges (be in the Administrators group, see here).
Obtaining data
I've provided two versions of the script. One which just dumps the entire output of the DMS table to a CSV file here and one which provides some processing of the data which can be used as an example here. I've chosen the table 'wls_webservice_operation' as sample but you might want to obtain data from a different table. How to do that is described below.
Selecting a different table
The Python script contains a DMS URL to obtain data from. For example: /dms/index.html?format=xml&cache=false&prefetch=false&table=wls_webservice_operation. You can open http://<host>:<port>/dms/index.html?format=xml&cache=false&prefetch=false&table=wls_webservice_operation in a webbrowser to see the results. If you want a nice HTML output, replace 'xml' in the url with 'metricstable'. The XML output in 11g and 12c has a different namespace but the script takes care of that for you.
If you have another table you want data from, you can go to the DMS Spy servlet page: http://<host>:<port>/dms and browse from there. If you see a specific table which has interesting data, open the link in a new tab and you can see the URL of the table. Next change the format to xml (metricstable is default when using the webinterface) and you have the URL which the script can use:
Processing data
The example script here already has some sorting and processing of data. This has been created by a colleague: Rudi Slomp. It contains specific fields to filter and sort. That's the following part:
After executing the script, the output could be something like:
Host;Operation;Service;MinTime;MaxTime;AvgTime;Calls
localhost;process;HelloWorld;13;1031;28,010460251046027;478
The Dutch Excel uses ',' instead of the usual '.' as decimal separator :(. For an English decimal separator replace:
with
If you select a different table, you need of course to change these fields also. So as in the sample of the above screenshots for the soainfra_composite table, you might only want to select the Name and soainfra_domain field. You might not require a selection and you might not require sorting. The above part would become something like:
The output would be a CSV (well, more like semicolon separated values) like:
Name;Partition
HelloWorld;mypartition
The Python script requires the 'requests' module to be installed. This module takes care of maintaining the DMS session. Installing this module can be challenge if the system you want to run the script from, has limited privileges or connectivity. Using WinPython might help. This distribution of Python is portable and can be copied standalone to a Windows folder or share. It will work just fine from there and is not dependant on Windows registry settings or specific OS configuration. I recommend you first prepare a WinPython folder with the requests module already installed and package that with your script if you have such a requirement.
set http_proxy=http://username:password@<host>:<port>
set https_proxy=https://username:password@<host>:<port>
After you have installed the module and the environment you want to connect to from the script, does not require you to go through the proxy server, do not forget to unset these.
set http_proxy=
set https_proxy=
Inside the script you should specify host, port, username and password of the WebLogic environment. The user should have administrative privileges (be in the Administrators group, see here).
Obtaining data
I've provided two versions of the script. One which just dumps the entire output of the DMS table to a CSV file here and one which provides some processing of the data which can be used as an example here. I've chosen the table 'wls_webservice_operation' as sample but you might want to obtain data from a different table. How to do that is described below.
The Python script contains a DMS URL to obtain data from. For example: /dms/index.html?format=xml&cache=false&prefetch=false&table=wls_webservice_operation. You can open http://<host>:<port>/dms/index.html?format=xml&cache=false&prefetch=false&table=wls_webservice_operation in a webbrowser to see the results. If you want a nice HTML output, replace 'xml' in the url with 'metricstable'. The XML output in 11g and 12c has a different namespace but the script takes care of that for you.
If you have another table you want data from, you can go to the DMS Spy servlet page: http://<host>:<port>/dms and browse from there. If you see a specific table which has interesting data, open the link in a new tab and you can see the URL of the table. Next change the format to xml (metricstable is default when using the webinterface) and you have the URL which the script can use:
Processing data
The example script here already has some sorting and processing of data. This has been created by a colleague: Rudi Slomp. It contains specific fields to filter and sort. That's the following part:
for row in rows:
columns = get_columns_from_row(row)
h,n,s,mint,maxt,avgt,comp='','','',0,0,0,0
for column in columns:
k,v = get_name_value_from_column(column)
if k == 'Host': h = v
if k == 'Name': n = v
if k == 'wls_ear': s = v
if k == 'Invoke.minTime': mint = exceldecimal(v)
if k == 'Invoke.maxTime': maxt = exceldecimal(v)
if k == 'Invoke.avg': avgt = exceldecimal(v)
if k == 'Invoke.completed': comp = int(v)
if comp > 0:
result.append([h,n,s,mint,maxt,avgt,comp])
result.sort(key=itemgetter(0)) #sort on wls_ear,name,host
result.sort(key=itemgetter(1))
result.sort(key=itemgetter(2))
result.insert(0, ['Host','Operation','Service','MinTime','MaxTime','AvgTime','Calls'])
After executing the script, the output could be something like:
Host;Operation;Service;MinTime;MaxTime;AvgTime;Calls
localhost;process;HelloWorld;13;1031;28,010460251046027;478
The Dutch Excel uses ',' instead of the usual '.' as decimal separator :(. For an English decimal separator replace:
if k == 'Invoke.minTime': mint = exceldecimal(v)
if k == 'Invoke.maxTime': maxt = exceldecimal(v)
if k == 'Invoke.avg': avgt = exceldecimal(v)
with
if k == 'Invoke.minTime': mint = v
if k == 'Invoke.maxTime': maxt = v
if k == 'Invoke.avg': avgt = v
If you select a different table, you need of course to change these fields also. So as in the sample of the above screenshots for the soainfra_composite table, you might only want to select the Name and soainfra_domain field. You might not require a selection and you might not require sorting. The above part would become something like:
for row in rows:
columns = get_columns_from_row(row)
n,d='',''
for column in columns:
k,v = get_name_value_from_column(column)
if k == 'Name':
n = v
if k == 'soainfra_domain':
d = v
result.append([n,d])
result.insert(0, ['Name','Partition'])
The output would be a CSV (well, more like semicolon separated values) like:
Name;Partition
HelloWorld;mypartition
What about IWS?
SOA Suite 12.2.1 comes with IWS, Integration Workload Statistics. Read more about it from my previous blog post here or from the official documentation here. IWS is part of the 'Oracle Integration Continuous Availability' option. This requires the 'Oracle WebLogic Server Continuous Availability' option. IWS is more specific to SOA/BPM Suite. Also statistics can be saved for later analysis and exported in a variety of formats from the EM Fusion Middleware Control. I've not spend time on automating gathering/scheduling/resetting IWS statistics from a script.
The DMS is more low-level and less specific for SOA/BPM Suite. You can for example also obtain detailed information on JVM behavior. Dynamic data like average service response times are reset upon server restart. The DMS gives way more information (also some very low level data) and requires careful selection of what you might want to know.
Depending on the change you want to performance-test, one might be more useful than the other. IWS is easier to use since it has UI support and does not require custom scripting. DMS might be a bit overwhelming at first due to the large amount of data which is available.
No comments:
Post a Comment