configuration
How many workers in such job? It is set by Guagua, user can set it, but it just a hint.
How many iterations or maximal iteration number. It is set by using '-c'.
Worker class, must implement ml.shifu.guagua.worker.WorkerComputable interface.
Master class, must implement ml.shifu.guagua.master.MasterComputable interface.
Master result class, must implement ml.shifu.guagua.io.Bytable interface or extend ml.shifu.guagua.io.HaltBytable class.
Worker result class, must implement ml.shifu.guagua.io.Bytable interface or extend ml.shifu.guagua.io.HaltBytable class.
User defined master interceptors, separated by ',' if you have multiple interceptor. All interceptors must implement ml.shifu.guagua.master.MasterInterceptor interface.
User defined worker interceptors, separated by ',' if you have multiple interceptor. All interceptors must implement ml.shifu.guagua.worker.WorkererInterceptor interface.
System master interceptors, default is 'ml.shifu.guagua.master.MasterTimer,ml.shifu.guagua.master.MemoryStatsMasterInterceptor,ml.shifu.guagua.master.SyncMasterCoordinator'
System worker interceptors, default is 'ml.shifu.guagua.worker.WorkerTimer,ml.shifu.guagua.worker.MemoryStatsWorkerInterceptor,ml.shifu.guagua.worker.SyncWorkerCoordinator'.
zookeeper servers. For example 'server1:port1,server2:port2'.
zookeeper session timeout, default is 5 * 60 * 1000 ms.
If zookeeper gets connection loss exception, max attempt number. By default is 5.
If zookeeper gets connection loss exception, waiting time mills before next connection.
Input folder.
Whether clean znodes resources when job ends, default is true.
Master serializer class to serialize master results, must implement interface ml.shifu.guagua.io.serializer By default is 'ml.shifu.guagua.io.BytableSerializer'.
Worker serializer class to serialize master results, must implement interface ml.shifu.guagua.io.serializer By default is 'ml.shifu.guagua.io.BytableSerializer'.
Whether support halt if master or worker get some condition like error converged. By default is true.
It is like pig, whether to combine splits together to get a number of byte set by 'guagua.split.maxCombinedSplitSize'.
Whether to enable zookeeper hearbeat, default is false since ZooKeeper client provided by zookeeper has heartbeat.
If one worker computation time is over this number three times. The task will be killed by itself and fail-over will start another task. This is useful to check slow nodes and make task run on another nodes. By default is 40s.
How many workers done in each iteration. If set it to 50%, half of all workers are done in one iteration and time is over 'guagua.min.workers.timeout', master will start another iteration without waiting for other un-ended tasks. By default is 1.
Below is Guagua-YARN related.
Guagua AppMaster container priority.
queue name set in Guagua-YARN.
Application name set by users.
Input folder, By using GuaguaYarnClient with '-i' the same with 'GuaguaMapReduceClient'.
Lib jars used in YARN tasks.
App jars used in YARN application master.
Guagua AppMaster memory, default is 1024 (MB).
Guagua AppMaster java command args.
App container task memory setting, by default is 1024(MB)
App container java command args.
Like 'mapred.mapper.max.attempts', by default is 4.
Guagua AppMaster vcores.
App container vcores.
A Netty-RPC server is started in AppMaster to accumulate info like progress. We can set the port of ther server.