JMX API
The purpose of the JMX API is to supply a JMX client the necessary functionality they have with a java implementation.
The JMX proxy is located in its own project under:
https://github.com/scylladb/scylla-jmx
It uses maven to compile, it contains a README file that describe how to compile and run it. But as it uses maven, it is simply:
mvn install
Note that you need to enable remote jmx for the nodetool to connect to it:
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7199 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -jar target/scylla-mbean-1.0.jar
In the service case, the configs are in /etc/sysconfig/scylla-jmx (you can enable the IP and remote connections to local service there - see SCYLLA_JMX_REMOTE and SCYLLA_API_ADDR )
By default the the JMX would connect to a node on the localhost on port 10000.
The jmx API uses the system properties to set the IP address and Port of the API it connect to. To change the ip address use the apiaddress property (e.g. -Dapiaddress=1.1.1.1) To change the port use the apiport (e.g. -Dapiport=10001)
The jmx API would try to use scylla configuration file to find the ip and port in the following locations:
- In the command line -Dapiconfig=file-name
- From the environment variable $SCYLLA_CONF/scylla.yaml
- From the environment variable $SCYLLA_HOME/conf/scylla.yaml
For the first phase, the management will be based on a RESTFull API and a java proxy.
- Should we support SSL for the the Rest API
A minimal java program that uses a Rest client to call seastar, it would expose MBean and would open the jmx connection to control it.
Migrate the httpserver from the OSv repository. The lower connection layer would be replaced with the httpd implementation.
The following functionality will be migrated:
- GET/POST message implementation
- Routes
- Swagger API doc
- Swagger Code generation
- Exception and error handling
- Json support
Use an existing C++ library with what other adaptations that are required to use C*. Alternative http server libraries
- libhttpserver - originally suggested by Pekka for an API rewrite.
- pion - A C++ boost based implementation
In this approach, there will be an integrated API for the server will be a standalone java application that will expose a JMX MBeans and that would query the system API. The java proxy can co-locate on the same machine or placed on a different node.
Run the jvm as a shared library from the C++ code, opening a JVM and use its JMX capabilities.
- RMI spec The java RMI spec can be found:
- JSR 160 JavaTM Management Extensions (JMX) Remote API/ JSR 3 JavaTM Management Extensions (JMXTM) Specification The JMX Spec
- JSR 262 Web Services Connector A JSR to connect over web service
- JSR 255 - Dormant JConsole uses by default either RMI/JRMP or RMI/JRMP over SSL.
nodetools support SSL as describe here: https://support.datastax.com/entries/43692547-Step-by-step-instructions-for-securing-JMX-authentication-for-nodetool-utility-OpsCenter-and-JConsol
RMI-IIOP specification The RMI protocol makes use of two other protocols for its on-the-wire format: Java Object Serialization and HTTP
The MBean server is the java entity that externally open a JMX interface and internally allows registering and manipulating MBean. "The MBean server is a registry for MBeans in the agent. The MBean server is the component that provides the services for manipulating MBeans. All management operations performed on the MBeans are done through the MBeanServer interface."
To implement the JMX in C++, we will need to act as an MBean server for external communication.
Retrieve a specific MBean by its object name.
- Retrieve a collection of MBeans, by means of pattern matching on their names, and optionally by means of a filter applied to their attribute values.
- Get one or several attribute value(s) of an MBean.
- Invoke an operation on an MBean.
- Discover the management interface of an MBean, that is, its attributes and operations. This is what is called the introspection of the MBean.
- Register interest in the notifications emitted by an MBean.
Session vs Connection
- A Session can have a state in the client, it does not necessarily have a state on the server. It can contains multiple connection.
- A connection on the extreme can be made per request (when using UDP for example)
Handshake messages:
- HandshakeBeginMessage
- HandshakeEndMessage
- HandshakeErrorMessage
- VersionMessage
Profile messages:
- TLSMessage (JMXMP Connector only)
- SASLMessage (JMXMP Connector only)
MBean server operation messages:
- MBeanServerRequestMessage
- MBeanServerResponseMessage
- NotificationRequestMessage
- NotificationResponseMessage
Connection messages
- CloseMessage
The JRMP is not really documented. Going over the code can be useful. The following class are part of the communication parts:
- DefaultMBeanServerInterceptor - This is the default class for MBean manipulation on the agent side. It contains the methods necessary for the creation, registration, and deletion of MBeans as well as the access methods for registered MBeans. This is the core component of the JMX infrastructure.
- RMIConnector - The RMI connector implementation
- RMIConnectionImpl - The RMI connection object
- MBeanServerDelegateImpl
- JmxMBeanServer - The default MbeanServer
- TCPTransport - Read from an input stream, the interesting parts starts at run0(), run the executeAcceptLoop that do the accept
- TCPConnection - Holds a tcp connection information holds the output stream for the connection
- TCPChannel -
- JMXConnectorFactory.connect(url, environment)
- Create RMIConnector
- RMIServerImpl.doNewClient -> RMIJRMPServerImpl.makeClient
- RMIConnector.connect(null)
- get server - RMIJRMPServerImpl
- connection = getConnection(server...) RMIConnectionImpl
- return RMIConnector
- Create RMIConnector
- RMIConnector.getMBeanServerConnection
- return RemoteMBeanServerConnection
- RemoteMBeanServerConnection.getDomains
- RMIJRMPServerImpl.
JMX security. Namely the Java Secure Socket Extension (JSSE), the Simple Authentication and Security Layer (SASL), and the Java Authentication and Authorization Service (JAAS)
The protocol is based on the java Object Serialization Stream Protocol. https://docs.oracle.com/javase/7/docs/platform/serialization/spec/protocol.html
- No security support
- no iiop support
More specific assumption in the estimation
- socket - either plain socket, or http stream.
- For httpstream - modify httpd code to support http POST and httpstream - 3 days
- TCPTransport - Protocol identifying and general command parsing 1 day, (Protocol can be one of three singleop, stream, multiplex, assumption multiplex is used)
- Need to support Ping & DGCAck - 1 day
- Call: (Transport.serviceCall & UnicastServerRef.dispatch) Assuming support for 1.2 version and up
- unmarshel method and parameters 3 days
- call the method (function/handler registry)
- marshel the results 3 days
- Possible auto code generation for creating serializer/deserializer from java classed - 3 days
The following java class, are part of the java code path, and are here as a reference, but are probably not needs to be implemented.
- StreamRemoteCall
- TCPEndpoint ep;
- TCPChannel ch;
- TCPConnection
- Handshake & connection messages 3 days
Profile messages
-
TLSMessage (JMXMP Connector only) will not be supported
-
SASLMessage (JMXMP Connector only) will not be supported
-
MBean server operation messages: MBeanServerRequestMessage, MBeanServerResponseMessage 2 days
-
NotificationRequestMessage will not be supported
-
NotificationResponseMessage will not be supported