Skip to content
This repository has been archived by the owner on May 6, 2020. It is now read-only.

Monitoring

Jump to bottom
mpermar edited this page Nov 8, 2011 · 7 revisions

Rayo Administration and Monitoring Interface

Rayo exposes an administration and monitoring interface based on Java MBeans. You can access these MBeans using any JMX enabled console and browse all the monitoring metrics that Rayo exposes (and also to run administrative tasks).

Rayo Server's administration and monitoring interface also features a JSON-over-HTTP interface that gives you access to all the metrics and all the administrative operations from platforms other than Java. You can also use this interface to build your own dashboards if you prefer.

Browsing the JMX data

All you need to do in order to browse the JMX information exposed by Rayo is run a JMX console application. The most popular one is jconsole, which is shipped with every JDK. Assuming you have a JDK installed and Java is in your PATH, you only need to go to your command line and run:

  $ jconsole

...and JConsole should pop up. JConsole's user interface is quite intuitive - you only need to select the process you want to monitor and the JMX dashboard will pop up. Choose the MBeans tab and browse to the com.rayo element in the JMX Beans tree at the left side of the dashboards. That's it, you can browse all the Rayo MBeans like in the picture below:

JConsole

I'd recommend JRockit's mission control, which comes as a binary tool bundled within JRockit's JDK and is a very powerful management console with a much nicer user interface. The drawback is you need to download the JRockit JDK and you need to run your application (i.e. you application server) using that JDK. This is a screenshot of Mission Control in action:

JConsole

Another nice option is VisualVM.

MBeans, attributes and operations

In the section below we explain the different MBeans and attributes exposed by Rayo.

Note that all the statistics and metrics exposed are not historical. They're tied to the lifespan of the Rayo application, so if your server gets restarted, all the counters will start from 0.

Application Information

The Info MBean exposes information about the application, such as the version number and build number.

JMX URL: com.rayo:Type=INFO

Attributes Exposed:

  • Version Number: Rayo Server version number.
  • Build Number: Build Number.
  • Build Id: Build Id. It includes the actual date of the build.

Call Statistics

The Call Statistics MBean exposes statistic information about all the calls and verbs handled by your Rayo server.

JMX URL: com.rayo:Type=CallStatistics

Attributes exposed:

  • Calls Accepted: Number of calls accepted.
  • Calls Answered: Number of calls answered.
  • Calls Busy: Number of calls that were busy.
  • Calls Hung Up: Number of calls that hung up.
  • Calls Failed: Number of calls that failed.
  • Calls Rejected: Number of calls that were rejected.
  • Calls Timed out: Number of calls that timed out.
  • Calls Ended Without Reason: Number of calls that ended with an unknown reason.
  • Calls Redirected: Number of calls that were redirected.
  • Calls Incoming: Number of incoming calls.
  • Calls Outgoing: Number of outgoing calls.
  • Verbs Created: Number of verbs created.

Calls

The Calls MBean offers information about all the active calls and verbs in your Rayo server. You are not only able to get counters about the active calls and verbs, but also access the actual live call and verb internal data.

JMX URL: com.rayo:Type=Calls

Attributes Exposed:

  • Active Verbs Count: Gives you the number of currently active verbs.
  • Active Calls Count: Gives you the number of currently active calls.
  • Active Calls: Gives you access to the data of all the currently live calls. For each call object you will get access to:
  • Id: Call Id.
  • CDR: The CDR for that call
  • Address: Call Address.
  • Attributes: Map with all call attributes.
  • Attributes String: A string representation for all the call attributes.
  • State: Call state.
  • Headers: Map with all the call headers.
  • Headers String: A string representation for all the call headers.
  • Invitee: Invitee's URI.
  • Invitor: Invitor's URI.
  • Peers: List with the URIs for all the peers for that call.
  • Verbs: List with all the active verbs for that call.
  • Accepted: Indicates whether the call has been accepted or not.
  • Processed: Indicates whether the call has been processed or not.
  • Redirected: Indicates whether the call has been redirected or not.
  • Rejected: Indicates whether the call has been rejected or not.
  • Supervised: Indicates whether the call is being supervised or not.
  • Active Verbs: Gives you access to the data of all the currently live verbs. For each verb object, you have access to:
  • Call: Call that initiated the verb.
  • Id: Verb's id.
  • Type: Verb's type.
  • Verb: ToString representation of the verb.
  • Complete: Whether the verb has been completed or not.

Rayo Statistics

The Rayo Statistics MBean exposes information about the Rayo subsystem. You get counters about the different XMPP stanzas that have been processed.

JMX URL: com.rayo:Type=Rayo

Attributes Exposed:

  • Calls Received: Number of Offer events received.
  • Call Events Processed: Number of events raised by the Moho layer.
  • IQs Received: Number of IQ stanzas received from the different clients.
  • Commands Count: A map that contains totals for all the different type of Rayo commands like 'say', 'ask', etc.
  • IQ Responses: Number of IQ responses received by Rayo.
  • IQ Errors: Number of IQ error stanzas sent to the different clients.
  • IQ Results: Number of IQ Result stanzas sent to the different clients.
  • Validation Errors: Number of invalid IQ messages received.
  • Message Stanzas Received: Number of XMPP message stanzas received.
  • Presence Stanzas Received: Number of XMPP presence stanzas received.

Mixer Statistics

The Mixer Statistics MBean exposes information about the Mixers

JMX URL: com.rayo:Type=MixerStatistics

Attributes Exposed:

  • ActiveMixersCount: Number of active mixers.
  • TotalMixers: Total number of mixers that have been created.

Admin

The Admin MBean exposes administrative operations that you can run in your Rayo Server.

JMX URL: com.rayo:Type=Admin,name=Admin

Attributes exposed:

  • Quiesce Mode: Current quiesce mode.

Operations exposed:

  • enableQuiesce(): Enables Quiesce mode. In this mode, no further messages and calls will be accepted.
  • disableQuiesce(): Disables Quiesce mode. This operation returns the Rayo server to a normal operation mode.
  • setLogLevel(logger,level): Allows you to alter the Log4j log level configuration for a specific logger at runtime. As a Logger, you can specify any package name like com.rayo, com.voxeo, com.rayo.server and so on. As level, you may use Log4j Log levels like: DEBUG, INFO, WARN, ERROR, etc.

CDR Mbeans

The CDRs MBean lets you access to all the active CDRs in the system.

JMX URL: com.rayo:Type=Cdrs

Attributes exposed:

  • ActiveCDRs: List all active CDRs in the system.

Depending on the CDR storage strategies that you are using (See CDR Support), you may also have access to extra Mbeans:

Filesystem-based CDR Storage configuration

JMX URL: com.rayo:Type=Admin,name=File CDR

Operations exposed:

  • changeFile(String): Changes the name of the file where CDRs are stored to.

XMPP PubSub based CDR Storage configuration

JMX URL: com.rayo:Type=Admin,name=Xmpp CDR

Operations exposed:

  • changeDestination(String server, Integer port, String username, String password, String node): Changes the settings for the XMPP PubSub node where CDRs are sent to.

AMQP CDR Storage configuration

JMX URL: com.rayo:Type=Admin,name=Amqp CDR

Operations exposed:

  • changeDestination(String server, Integer port, String username, String password, String exchange, String route): Changes the settings for the AMQP Server where CDRs are sent to.

JSON Interface

The Rayo Administration and Monitoring JSON interface is available from the /jmx path. If your Rayo installation is available at http://localhost:8080/rayo, then you should run the following:

  $ curl http://localhost:8080/rayo/jmx

  {"timestamp":1306490784,"status":200,"request":{"type":"version"},"value":{"protocol":"4.2","agent":"0.90","info":{}}}

Getting access to the metrics

To access any of the exposed JMX attributes, you can simply send a GET or POST request with the read command along with the actual JMX URL that you want to access. For example, if you want to get access to the build information:

  $ curl http://localhost:8080/rayo/jmx/read/com.rayo:Type=Info

  {"timestamp":1306918843,"status":200,"request":{"mbean":"com.rayo:Type=Info","type":"read"},"value":{"BuildNumber":145,"VersionNumber":"2.0-SNAPSHOT","BuildId":"2011-06-01_08-51-13"}}martin@martin-ThinkPad-W500:~/Workspace-2.6.0/sipmethod-10/_dist/core/standard/bin$ 2011-06-01 11:00:43.681 DEBUG [http-0.0.0.0-8080-exec-1] #HTTP#: (o)HTTP/1.1 200\r\nContent-Type: text/plain;charset=utf-8\r\nContent-Length: 184\r\nDate: Wed, 01 Jun 2011 09:00:43 GMT\r\n\r\n

Or if you want to get access to the call statistics:

  $ curl http://localhost:8080/rayo/jmx/read/com.rayo:Type=CallStatistics

  {"timestamp":1306491113,"status":200,"request":{"mbean":"com.rayo:Type=CallStatistics","type":"read"},"value" {"IncomingCalls":3,"CallsAnswered":0,"CallsFailed":2,"CallsAccepted":0,"CallsBusy":0,"CallsRejected":0,"TotalCalls":3,"CallsRedirected":0,"CallsTimedout":0,"OutgoingCalls":0,"CallsHangedUp":0,"CallsEndedUnknownReason":0,"TotalVerbs":0}}

To get access to a specific attribute, just add it to the path:

  $ curl http://localhost:8080/rayo/jmx/read/com.rayo:Type=CallStatistics/IncomingCalls

  {"timestamp":1306491266,"status":200,"request":{"mbean":"com.rayo:Type=CallStatistics","attribute":"IncomingCalls","type":"read"},"value":3}

As you have already seen from the attibutes description section above, Rayo exposes arrays of more complex objects like live calls and verbs. To get access to one of these arrays, you just use the name:

  $curl http://localhost:8080/rayo/jmx/read/com.rayo:Type=Calls/ActiveCalls

  {"timestamp":1306491889,"status":200,"request":{"mbean":"com.rayo:Type=Calls","attribute":"ActiveCalls","type":"read"},"value":[{"headers":{"[Allow]":{"value":"INVITE","key":"Allow"},"[Max-Forwards]":{"value":"70","key":"Max-Forwards"},"[Subject]":{"value":"Phone call","key":"Subject"},"[To]":{"value":"PrismLocal <sip:userc@127.0.0.1:5060>","key":"To"},"[Content-Length]":{"value":"400","key":"Content-Length"},"[Contact]":{"value":"<sip:martin@127.0.0.1:6089>","key":"Contact"},"[Via]":{"value":"SIP\/2.0\/UDP 127.0.0.1:6089;rport=6089;branch=z9hG4bK1188342661","key":"Via"},"[User-Agent]":{"value":"Linphone\/3.3.2 (eXosip2\/3.3.0)","key":"User-Agent"},"[CSeq]":{"value":"20 INVITE","key":"CSeq"},"[Content-Type]":{"value":"application\/sdp","key":"Content-Type"},"[Call-ID]":{"value":"160283831","key":"Call-ID"},"[From]":{"value":"<sip:martin@192.168.1.33:6089>;tag=1360952042","key":"From"}},"callState":"ACCEPTED","address":"sip:martin@192.168.1.33:6089","invitor":"sip:martin@192.168.1.33:6089","peers":[],"verbs":[],"invitee":"sip:userc@127.0.0.1:5060","attributes":"[]","callId":"b05db3ae-4aaf-46a3-ad1e-7730cea59b2b"}]}

If you want to access one of the elements of the array, then you only need to add the index to the path:

  $ curl http://localhost:8080/rayo/jmx/read/com.rayo:Type=Calls/ActiveCalls/0/

  {"timestamp":1306491938,"status":200,"request":{"mbean":"com.rayo:Type=Calls","path":"0","attribute":"ActiveCalls","type":"read"},"value":{"headers":{"[Allow]":{"value":"INVITE","key":"Allow"},"[Max-Forwards]":{"value":"70","key":"Max-Forwards"},"[Subject]":{"value":"Phone call","key":"Subject"},"[To]":{"value":"PrismLocal <sip:userc@127.0.0.1:5060>","key":"To"},"[Content-Length]":{"value":"400","key":"Content-Length"},"[Contact]":{"value":"<sip:martin@127.0.0.1:6089>","key":"Contact"},"[Via]":{"value":"SIP\/2.0\/UDP 127.0.0.1:6089;rport=6089;branch=z9hG4bK1188342661","key":"Via"},"[User-Agent]":{"value":"Linphone\/3.3.2 (eXosip2\/3.3.0)","key":"User-Agent"},"[CSeq]":{"value":"20 INVITE","key":"CSeq"},"[Content-Type]":{"value":"application\/sdp","key":"Content-Type"},"[Call-ID]":{"value":"160283831","key":"Call-ID"},"[From]":{"value":"<sip:martin@192.168.1.33:6089>;tag=1360952042","key":"From"}},"callState":"ACCEPTED","address":"sip:martin@192.168.1.33:6089","invitor":"sip:martin@192.168.1.33:6089","peers":[],"verbs":[],"invitee":"sip:userc@127.0.0.1:5060","attributes":"[]","callId":"b05db3ae-4aaf-46a3-ad1e-7730cea59b2b"}}

Then to access any attribute of that array element you proceed as usual:

  $ curl http://localhost:8080/rayo/jmx/read/com.rayo:Type=Calls/ActiveCalls/0/callState

  {"timestamp":1306492020,"status":200,"request":{"mbean":"com.rayo:Type=Calls","path":"0\ callState","attribute":"ActiveCalls","type":"read"},"value":"ACCEPTED"}

Executing operations

To execute operations, you use the exec path. For example, to enable quiesce mode we would run:

  $ curl http://localhost:8080/rayo/jmx/exec/com.rayo:Type=Admin,name=Admin/enableQuiesce

  {"timestamp":1306492257,"status":200,"request":{"operation":"enableQuiesce","mbean":"com.rayo:Type=Admin,name=Admin","type":"exec"},"value":null}

Then to disable it:

  $ curl http://localhost:8080/rayo/jmx/exec/com.rayo:Type=Admin,name=Admin/disableQuiesce

  {"timestamp":1306492296,"status":200,"request":{"operation":"disableQuiesce","mbean":"com.rayo:Type=Admin,name=Admin","type":"exec"},"value":null}

Executing operations with arguments

To pass any arguments to a MBean operation, you only need to append the required parameters to the HTTP path. For example, if you wish to change the Log4j log level for the Logger com.rayo to DEBUG you only need to run the following:

$ curl http://localhost:8080/rayo/jmx/exec/com.rayo:Type=Admin,name=Admin/setLogLevel/com.rayo/debug

Security

Security is controlled with the jolokia-access.xml configuration file. In that file, you can configure which hosts will have access to the MBeans, which operations may be executed, which HTTP methods are supported or you can also explicitly allow or deny access to specific MBeans.

The security constraints are manged with Jolokia. Feel free to check the security documentation here.

Extending the documentation

That was easy, wasn't it? Most of the credit should go to the guys who created the Jolokia library, which is an Open Source library for remote JMX access. If you browse their documentation you will find tons of different examples, including particularly useful ones about how to use the JSON API. There you will find information about sorting escaping issues, how to use POST requests instead of GET requests, how to search for specific MBeans, how to list MBeans, how to enable security and many other topics.

Admin Dashboard

Rayo also features an administration dashboard that you can use to track your MBeans. This dashboard uses Jolokia's JavaScript client library, JQuery and HighCharts to create a nice and easy to configure dashboard interface that lets you create your own management dashboards in a matter of minutes. Here is a sample dashboard:

Sample Dashboard

Dashboards are easily configurable by just adding small chunks of code to a JavaScript configuration file. Rayo comes with a very simple default dashboard that you can view by browsing to the URL http://YOUR_SERVER/rayo. That dashboard can be configured by editing the jmx.js configuration file.

For example, adding a new chart that shows two line graphs with the actual memory usage of your server in the dashboard is as easy as writing a few lines of JavaScript code:

	factory.create([
		{
			name: 'java.lang:type=Memory',
			attribute: 'HeapMemoryUsage',
			path: 'committed'
		},
		{
			name: 'java.lang:type=Memory',
			attribute: 'HeapMemoryUsage',
			path: 'used'
		}
	]);

Log4j configuration

When you set Prism's log4j to a DEBUG level then the amount of HTTP requests that a dashboard makes to Prism (one per second) can create a really verbose sipmethod.log log file which makes very difficult to find anything on it. One workaround for this issue is to edit the $PRISM_HOME/conf/log4j.properties file and add the following lines to it:

log4j.logger.com.micromethod.sipmethod.server.http=INFO 
log4j.logger.com.micromethod.sipmethod.server.container.impl.http=INFO 
log4j.logger.com.micromethod.sipmethod.server.container.impl.filter=INFO

which will disable DEBUG log level for the packages above.

Clone this wiki locally