How do I run a cluster on Linux（centos） #285

wuyanxing · 2019-03-27T03:18:53Z

No description provided.

wuyanxing · 2019-03-27T03:19:36Z

I can't find any explanation or method.

jkliss · 2019-05-20T09:01:48Z

I am having the same problem.
As I understood from the documentation it should be sufficient to add servers to the config file.
GE then at least sends messages between these servers but it does not run the script. The script was running successfully in embedded mode. I also used Global.CloudStorage.SaveXYZ to access memory. What can I do to set it up correctly?

This is the config I was trying to use:

<Trinity ConfigVersion="2.0">
  <Local>
    <!-- Add any configuration the client might need -->
  </Local>

    <section name="Application">
     <entry name="ConfigOutputOn">true</entry>
     <entry name="CurrentRunningMode">Distributed</entry>
    </section>

    <section name="Network">
     <entry name="ClientBufferSize">1048576</entry>
     <entry name="ClientMaxBufferSize">134217728</entry>
     <entry name="ServerSocketBufferSize">8192</entry>
     <entry name="ClientSocketBufferSize">8192</entry>
     <entry name="ServerMaxConn">512</entry>
     <entry name="ServerMaxAcceptOps">512</entry>
     <entry name="PreferedNetworkMask"/>
    </section>

    <Cluster>
        <Server Endpoint="<IPADRESS_SERVER1>:8133" />
        <Server Endpoint="<IPADRESS_SERVER2>:8133" />
    </Cluster>
</Trinity>

yatli · 2019-05-20T09:36:03Z

@jkliss could you attach the log for the two instances?

jkliss · 2019-05-20T11:25:13Z

[ INFO    ] EchoOnConsole set to ON
[ INFO    ] Log: changing logging directory to /home/jkliss/BFS/BFSClient/bin/Debug/netcoreapp2.2/trinity-log.
[ INFO    ] Loading Graph Engine Extensions.
[ INFO    ] Scanning for TSL storage extension.
[ INFO    ] TSL storage extension loaded.
[ INFO    ] Scanning for MemoryCloud extensions.
[ INFO    ] No MemoryCloud extension found.
[ INFO    ] Scanning for startup tasks.
[ INFO    ] EventLoop: Starting.
[ INFO    ] *****************************************************
[ INFO    ] ServerCount: 2
[ INFO    ]     IPADRESS_SERVER1:8133
[ INFO    ]     IPADRESS_SERVER2:8133
[ INFO    ] ProxyCount: 0
[ INFO    ] *****************************************************

if I replace IPADRESS_SERVER1 with localhost I get the following additional lines (and it starts sending network packets but the script doesn't go on further):

[ INFO    ] LocalMemoryStorage is initialized in read-write mode
[ INFO    ] Initializing logging facility
[ INFO    ] Reading log file.
[ INFO    ] Write-ahead-log successfully loaded. Recovered 0 records.
[ INFO    ] Creating write-ahead log file /home/jkliss/BFS/BFSClient/bin/Debug/netcoreapp2.2/storage/B/write_ahead_log/primary_storage_log_20.dat

jkliss · 2019-05-20T13:08:42Z

@yatli is there anything that I need to incorporate in the code to enable communication between servers or is there a function to make a server wait for another server to make them synchronize?

If you have an example on how to setup and run a distributed system on linux it would be very helpful for me and probably for others too

ToxicJojo · 2019-05-20T13:16:52Z

I'm having the same problem as @jkliss . Documentation on how to setup GraphEngine in a distributed way would be very useful.

edouardpoitras · 2023-08-30T18:30:32Z

I'm also looking for a working example of a distributed GraphEngine with multiple servers.

TaviTruman · 2023-08-30T22:57:41Z

@edouardpoitras Hi. Here is an example of how to configure for Graph Engine Availability Group (Cluster): This is my xml configuration on my head server (Only)

<!--Declare and Define the Head (Primary) Graph Engine Cluster-->
<!-- <Local Template="primary-rub-truespark-sf-cluster-template"/>-->
<!-- <Remote Template="rub-truespark-ontology-taxonomy-cluster-template"/>-->

<!--A Cluster node contains configurations for servers and proxies of a Graph Engine cluster. 
    There can be multiple Cluster nodes as long as they have different identifiers. 
			        Endpoint="10.1.10.5:7001" 
    A Cluster node can have an optional attribute Id.-->

<Cluster RunningMode="Server">
	<!-- IKW-GE-APPSVR03.inknowworksdev.net -->
	<Server Endpoint="10.1.100.5:7001" AssemblyPath="D:\Trinity TripleStore Server Deployment\">
		<Network HttpPort="-1" 
				CleintMaxConn="2" 
			    ClientSendRetry="5" 
			    ClientReconnectRetry="5" 
			    Handshake="TRUE"/>

		<Logging LogDirectory="D:\GraphEngine-Log\"
				LogLevel="Verbose" 
				LogToFile="TRUE" 
             	EchoOnConsole="TRUE"/>

		<Storage TrunkCount="256"
				ReadOnly="FALSE" 
				StorageCapacity="Max16G" 
				StorageRoot="D:\GraphEngine-Storage\" 
				DefragInterval="600"/>

		<LIKQ Timeout="90000" />
	</Server>

	<!-- IKW-GE-APPSVR03.inknowworksdev.net -->
	<Server Endpoint="10.30.10.23:7001" AssemblyPath="D:\Trinity TripleStore Server Deployment\">
		<Network HttpPort="-1" 
				CleintMaxConn="2" 
			    ClientSendRetry="5" 
			    ClientReconnectRetry="5" 
			    Handshake="TRUE"/>

		<Logging LogDirectory="D:\GraphEngine-Log\"
				LogLevel="Verbose" 
				LogToFile="TRUE" 
             	EchoOnConsole="TRUE"/>

		<Storage TrunkCount="256"
				ReadOnly="FALSE" 
				StorageCapacity="Max16G" 
				StorageRoot="D:\GraphEngine-Storage\" 
				DefragInterval="600"/>

		<LIKQ Timeout="90000" />
	</Server>

	<!-- IKW-GE-APPSVR03.inknowworksdev.net -->
	<Server Endpoint="10.30.10.34:7001" AssemblyPath="D:\Trinity TripleStore Server Deployment\">
		<Network HttpPort="-1" 
				CleintMaxConn="2" 
			    ClientSendRetry="5" 
			    ClientReconnectRetry="5" 
			    Handshake="TRUE"/>

		<Logging LogDirectory="D:\GraphEngine-Log\"
				LogLevel="Verbose" 
				LogToFile="TRUE" 
             	EchoOnConsole="TRUE"/>

		<Storage TrunkCount="256"
				ReadOnly="FALSE" 
				StorageCapacity="Max16G" 
				StorageRoot="D:\GraphEngine-Storage\" 
				DefragInterval="600"/>

		<LIKQ Timeout="90000" />
	</Server>

	<!-- IKW-GE-APPSVR03.inknowworksdev.net -->
	<Server Endpoint="10.30.10.108:7002" AssemblyPath="D:\Trinity TripleStore Server Deployment\">
		<Network HttpPort="-1" 
				CleintMaxConn="2" 
			    ClientSendRetry="5" 
			    ClientReconnectRetry="5" 
			    Handshake="TRUE"/>

		<Logging LogDirectory="D:\GraphEngine-Log\"
				LogLevel="Verbose" 
				LogToFile="TRUE" 
             	EchoOnConsole="TRUE"/>

		<Storage TrunkCount="256"
				ReadOnly="FALSE" 
				StorageCapacity="Max16G" 
				StorageRoot="D:\GraphEngine-Storage\" 
				DefragInterval="600"/>

		<LIKQ Timeout="90000" />
	</Server>

	<!-- IKW-GE-APPSVR03.inknowworksdev.net -->
	<Server Endpoint="10.30.10.71:7002" AssemblyPath="D:\Trinity TripleStore Server Deployment\">
		<Network HttpPort="-1" 
				CleintMaxConn="2" 
			    ClientSendRetry="5" 
			    ClientReconnectRetry="5" 
			    Handshake="TRUE"/>

		<Logging LogDirectory="D:\GraphEngine-Log\"
				LogLevel="Verbose" 
				LogToFile="TRUE" 
             	EchoOnConsole="TRUE"/>

		<Storage TrunkCount="256"
				ReadOnly="FALSE" 
				StorageCapacity="Max16G" 
				StorageRoot="D:\GraphEngine-Storage\" 
				DefragInterval="600"/>

		<LIKQ Timeout="90000" />
	</Server>
</Cluster>

TaviTruman · 2023-08-30T23:08:58Z

For the other machines in the GE Cluster you only need to specify the local machine:

<!--Declare and Define the Head (Primary) Graph Engine Cluster-->
<!-- <Local Template="primary-rub-truespark-sf-cluster-template"/>-->
<!-- <Remote Template="rub-truespark-ontology-taxonomy-cluster-template"/>-->

<!--A Cluster node contains configurations for servers and proxies of a Graph Engine cluster. 
    There can be multiple Cluster nodes as long as they have different identifiers. 
			        Endpoint="10.1.10.5:7001" 
    A Cluster node can have an optional attribute Id.-->
<Cluster RunningMode="Server">
	<!-- IKW-GE-APPSVR03.inknowworksdev.net -->
	<Server Endpoint="10.30.10.23:7001" AssemblyPath="D:\Trinity TripleStore Server Deployment\">
		<Network HttpPort="-1" 
			CleintMaxConn="2" 
			ClientSendRetry="5" 
			ClientReconnectRetry="5" 
			Handshake="TRUE"/>

		<Logging LogDirectory="D:\GraphEngine-Log\"
			LogLevel="Verbose" 
			LogToFile="TRUE"
			EchoOnConsole="TRUE"/>

		<Storage TrunkCount="256"
			ReadOnly="FALSE" 
			StorageCapacity="Max16G" 
			StorageRoot="D:\GraphEngine-Storage\" 
			DefragInterval="600"/>

		<LIKQ Timeout="90000" />
	</Server>
</Cluster>

edouardpoitras · 2023-08-31T20:41:36Z

Hey @TaviTruman, thanks for the quick reply.

I've tried your configs and am still scratching my head :)

To simplify things, I have three servers in the cluster (127.0.0.1:700[0-2]). I've stripped out most of the config options but the endpoint values.

I run the first "head" node with the cluster config:

...
[ INFO    ] My IPEndPoint: 127.0.0.1:7000
...
[ INFO    ] ServerCount: 3
[ INFO    ]     127.0.0.1:7000
[ INFO    ]     127.0.0.1:7001
[ INFO    ]     127.0.0.1:7002
[ INFO    ] ProxyCount: 0
...

Great!

Then I run a 2nd instance with a config that only contains the server 127.0.0.1:7001:

[ INFO    ] My IPEndPoint: 127.0.0.1:7001
...
[ INFO    ] ServerCount: 1
[ INFO    ]     127.0.0.1:7001
[ INFO    ] ProxyCount: 0

And then the last node:

[ INFO    ] My IPEndPoint: 127.0.0.1:7002
[ INFO    ] ServerCount: 1
[ INFO    ]     127.0.0.1:7002
[ INFO    ] ProxyCount: 0

At this point, all three instances have reported:

...
[ INFO    ] Scanning for MemoryCloud extensions.
[ INFO    ] No MemoryCloud extension found.
...
[ INFO    ] Server 0 is successfully started.

Is that correct? No sure how to get the MemoryCloud extension working...

Also, when running a client, I re-used the first cluster config and then in the code specified: TrinityConfig.CurrentRunningMode = RunningMode.Client;
That seems to work in that the DistributedHashtable sample works. But it's unclear to me if it's working as expected.

Thanks again for you help!

TaviTruman · 2023-08-31T21:39:14Z

@edouardpoitras Hi. Looks like you've made great progress. If you don't mind please upload your trinity.xml configure file and your GE server and Client code. I'll get back you asap. Looks like you are trying to run the Distributed hash demo code, right.

edouardpoitras · 2023-09-01T14:07:43Z

Hey @TaviTruman, thanks again for your time. I created a new repository to create a working minimal cluster example: https://github.com/edouardpoitras/TrinityResearch

It's basically the DistributedHashtable sample with a few code tweaks in the Program.cs. Nothing changed in the DistributedHashtableServer.cs or DistributedHashtable.tsl.

The README.md file has the manual steps I'm trying. I want to eventually get this working with docker-compose.

I've tweaked a few things since we last talked. The closest I've gotten now is to use the same config but shuffle the server endpoint definitions around so the first one chosen is different for each config. Pretty sure I'm doing something wrong.

Let me know what you think/spot.

TaviTruman · 2023-09-01T16:09:04Z

Hi, @edouardpoitras, I will take a look at this today and get back to you shortly.

TaviTruman · 2023-09-04T19:20:27Z

I will upload a diagram that depicts the GE Cluster along with the trinity configuration for each server; the GE Client configuration is quite simple and typically all that you need to specify TrinityConfig.CurrentRunningMode = RunningMode.Client;

TaviTruman · 2023-09-05T06:03:57Z

Hi, @edouardpoitras. I have written a new and improved version of the "Distributed Hash Table" sample, and it works well in the GE Availability Group of three servers. It is late here so I will post it for you in the morning. I have written a new set of documentation as well describing the API sets.

TaviTruman · 2023-09-05T06:04:10Z

Hi, @edouardpoitras. I have written a new and improved version of the "Distributed Hash Table" sample, and it works well in the GE Availability Group of three servers. It is late here so I will post it for you in the morning.

edouardpoitras · 2023-09-05T13:31:42Z

@TaviTruman excellent! Looking forward to it 👍

TaviTruman · 2023-09-05T14:57:25Z

TaviTruman · 2023-09-05T14:59:16Z

GE DHT Server (Head): Running on my Windows 11 Desktop

TaviTruman · 2023-09-05T15:01:45Z

GE DHT Server (Secondary Server in GE Cluster): Running in Hyper-V Windows Server 2022

In the GE log you can see that some of the work has been distributed to this DHT Server Instance

TaviTruman · 2023-09-05T15:03:23Z

GE DHT Server (Secondary Server in GE Cluster): Running in Hyper-V Windows Server 2022. This is the 3rd server in the GE Cluster

You can see here that this server in my test never received any work.

TaviTruman · 2023-09-05T15:04:58Z

Here is my DHT Client doing work!

TaviTruman · 2023-09-05T15:13:31Z

Here are the trinity.xml config files:

GE DHT Server Cluster Config (see attached file)
2nd GE DHT Server Instance config
3rd GE DHT Server Instance config
primary GE DHT Cluster trinity.zip

TaviTruman · 2023-09-05T15:20:04Z

I will create PR and submit the new demonstration. In the meantime, I will upload the VS Studio project for you here. FYI, I do have a new Discord Channel coming up in October; the channel is dedicated to all things Graph Engine, Knowledge Graphs, Ontology Driven Software Design (ODSD) using Graph Engine, and much, much more.

TaviTruman · 2023-09-05T15:22:18Z

Here is the VS Solution and Project structure I use:

TaviTruman · 2023-09-05T15:23:23Z

Here you go!

Distributed Hash Table on GE Cluster.zip

edouardpoitras · 2023-09-05T17:23:19Z

Amazing @TaviTruman - giving it a shot now.
I noticed that the projects use GraphEngine v4.0 - that doesn't seem available in the repo.
I tried changing the references to use v3.0 instead, but I get a Unable to find package GraphEngine.Client error. Presumably GraphEngine.Client is new in v4?

Thanks again!

Edit: Looking over the code and configs - this is exactly what I was looking for and is making a lot of sense to me now. Just need to work out how to get v4.

TaviTruman · 2023-09-05T17:33:28Z

Yikes - I forgot about that - sorry! I maintain my own public repo of the Graph Engine and have a lot of updates. Let me upload the new Nuget packages for you. FYI - the GraphEngine.Client was removed from this repo, but I have been using it for a few years now.

TaviTruman · 2023-09-05T17:36:29Z

Let me get you everything you need and then you can have a lot of fun :-) I have also updated the code so that it saves the Local Memory Cloud and then restores or reloads it.
Distributed Hash Table on GE Cluster -V2.zip

TaviTruman · 2023-09-05T17:38:41Z

Here is the Windows Ready Deployment

DHT server Deployment.zip

TaviTruman · 2023-09-05T17:42:45Z

Here are the Nuget Packages from my 4.x local builds. I will update my public GE Git Repo later today and can use it.

GraphEngine.Client.4.0.12049.symbols.zip

TaviTruman · 2023-09-05T17:44:50Z

Oh, you are using Linux, right?

libTrinity.zip

TaviTruman · 2023-09-05T17:47:17Z

Here is the link to my repo: https://github.com/InKnowWorks/IKW-GraphEngine

There is a lot of new work here - most of it you may not need.

edouardpoitras · 2023-09-05T19:41:15Z

Success! Thank you very much @TaviTruman, I was able to get it working on Linux as per your screenshots. I really like your project structure and will convert my repo to use that as well. I'm still going to try to get a minimal version working with Docker + compose in my TrinityResearch repo as I think that could be useful for others, but the real gold here is in your comments above 👍

Hopefully this thread will save headaches for others in the future.

TaviTruman · 2023-09-05T20:32:53Z

FYI - I am working out the wrinkles for Azure Kubernetes Deployment. I will post it in the Discussion as soon as it is bullet-proof.

wuyanxing changed the title ~~How do I run a cluster on Linux~~ How do I run a cluster on Linux（centos） Mar 27, 2019

How do I run a cluster on Linux（centos） #285

How do I run a cluster on Linux（centos） #285

Comments

wuyanxing commented Mar 27, 2019

wuyanxing commented Mar 27, 2019

jkliss commented May 20, 2019 • edited

yatli commented May 20, 2019

jkliss commented May 20, 2019

jkliss commented May 20, 2019

ToxicJojo commented May 20, 2019

edouardpoitras commented Aug 30, 2023

TaviTruman commented Aug 30, 2023 • edited

TaviTruman commented Aug 30, 2023 • edited

edouardpoitras commented Aug 31, 2023 • edited

TaviTruman commented Aug 31, 2023

edouardpoitras commented Sep 1, 2023

TaviTruman commented Sep 1, 2023

TaviTruman commented Sep 4, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

edouardpoitras commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023 • edited

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

edouardpoitras commented Sep 5, 2023 • edited

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

TaviTruman commented Sep 5, 2023 • edited

edouardpoitras commented Sep 5, 2023

TaviTruman commented Sep 5, 2023

jkliss commented May 20, 2019 •

edited

TaviTruman commented Aug 30, 2023 •

edited

TaviTruman commented Aug 30, 2023 •

edited

edouardpoitras commented Aug 31, 2023 •

edited

TaviTruman commented Sep 5, 2023 •

edited

edouardpoitras commented Sep 5, 2023 •

edited

TaviTruman commented Sep 5, 2023 •

edited