/
testing.txt
208 lines (178 loc) · 15.8 KB
/
testing.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
python -m da --logfile --logfilename logs\log.txt -F info --message-buffer-size 100000 src\bcr.da config1.txt
a file named testing.txt with an entry for each test case.
each entry should include:
(1) brief description of the scenario (i.e., the client requests, failure scenario, etc.),
(2) the name of the configuration file used to run the test case, [2017-10-16] revised next item]
(3) other information (if any) needed to run the test case, including the commands(s) used to run it (or the name of a shell script containing those commands) and any requirements such as that certain files should contain certain content,
(4) the name of the log file produced by running the test case, [2017-10-14 added next item],
(5) description of the programmatic check of correctness of content of dictionary (if any),
(6) the outcome (pass or fail), with a brief explanation of the problem in case of failure.
Test case 1: verify_result_single_client_2
(1) only one client having workload-
workload[0] = put('movie','star'); append('movie',' wars'); slice('movie','3:6'); get('movie')
t = 2
(2) configuration file- config_result_verify_testcase.txt
(3) python -m da --logfile -n MainNode --logfilename logs\log_verify_result_single_client_2_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_result_verify_testcase_2.txt
python -m da --logfile -n ClientNode --logfilename logs\log_verify_result_single_client_2_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_result_verify_testcase_2.txt
(4) log files
logs\log_verify_result_single_client_2_mainnode.txt
logs\log_verify_result_single_client_2_clientnode.txt
(5) A state object is maintained at client and same operations are performed on it as the operations requested
from the replica. The result for operation on dictionary at client is compared with result received from replica.
The count of correct results received is maintained at client and if it is 4 the result is PASS otherwise it is FAIL.
Also after the test case the state object from the head is requested and its content is compared with the content
of the object at client.
(6) outcome- PASS
Test case 2: client_timeout_testcase_2
(1) only one client having workload
workload[0] = put('movie','star'); append('movie',' wars'); slice('movie','3:6'); get('movie')
client timeout at 0ms. t = 2
(2) configuration file- config_client_timeout_testcase_2.txt
(3) python -m da -n MainNode --logfile --logfilename logs\log_client_timeout_2_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_client_timeout_testcase_2.txt
python -m da -n ClientNode --logfile --logfilename logs\log_client_timeout_2_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_client_timeout_testcase_2.txt
(4) log files-
logs\log_client_timeout_2_mainnode.txt
logs\log_client_timeout_2_clientnode.txt
(5) a dictionary is maintained at client. The result for operation on dictionary at client
is compared with result received from replica. The count of correct results received is
maintained at client and if it is 4 the result is PASS otherwise it is FAIL.
(6) outcome- PASS
Test case 3: verify_result_single_client_pseudorandom_testcase_1
(1) only one client having workload = pseudorandom(255,250)
(2) configuration file- config_pseudorandom_result_verify_testcase_1.txt
(3) python -m da -n MainNode --logfile --logfilename logs\log_verify_result_single_client_pseudorandom_testcase_1_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_result_verify_pseudorandom_testcase_1.txt
python -m da -n ClientNode --logfile --logfilename logs\log_verify_result_single_client_pseudorandom_testcase_1_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_result_verify_pseudorandom_testcase_1.txt
(4) log files-
logs\logs\log_verify_result_single_client_pseudorandom_testcase_1_clientnode.txt
log\log_verify_result_single_client_pseudorandom_testcase_1_mainnode.txt
(5) A state object is maintained at client and same operations are performed on it as the operations requested
from the replica. The result for operation on dictionary at client is compared with result received from replica.
The count of correct results received is maintained at client and if it is 250 the result is PASS otherwise it is FAIL.
Also after the test case the state object from the head is requested and its content is compared with the content
of the object at client.
(6) outcome- PASS
Test case 4: verify_result_multi_client_1
(1) two clients
workload[0] = put('movie','star'); append('movie',' wars'); slice('movie','3:6'); get('movie')
workload[1] = put('jedi,'luke skywalker'); get('jedi')
(2) configuration file- config_multiclient_result_verify_testcase_1.txt
(3) python -m da -n MainNode --logfile --logfilename logs\log_config_multiclient_result_verify_testcase_1_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_multiclient_result_verify_testcase_1.txt
python -m da -n ClientNode --logfile --logfilename logs\log_config_multiclient_result_verify_testcase_1_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_multiclient_result_verify_testcase_1.txt
(4) log file-
logs\log_config_multiclient_result_verify_testcase_1_clientnode.txt
logs\log_config_multiclient_result_verify_testcase_1_mainnode.txt
(5) A state object is maintained at client and same operations are performed on it as the operations requested
from the replica. The result for operation on dictionary at client is compared with result received from replica.
The count of correct results received is maintained at client and if it is 4 the result is PASS otherwise it is FAIL.
Also after the test case the state object from the head is requested and its content is compared with the content
of the object at client.
(6) outcome- PASS
Test case 5: stress_test
(1) 4 clients each having workload= pseudorandom(255,100)
(2) configuration file- config_stress_test.txt
(3) python -m da -n MainNode --logfile --logfilename logs\log_stress_test_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_stress_test.txt
python -m da -n ClientNode --logfile --logfilename logs\log_stress_test_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_stress_test.txt
(4) log files-
logs\logs\log_stress_test_mainnode.txt
logs\log_stress_test_clientnode.txt
(5) outcome- PASS
Test Case 6:
(1) 2 clients having workload of pseudorandom(233,6) and put('jedi,'luke skywalker'); slice('jedi','0:4'); get('jedi'); put('movie', 'wars'); get('movie2') respectively.
There are 2 failures injected in this scenario -
1st one triggers at replica 1 and occurs when client 0 sends its 6th request and injects failure of changing result to hash of OK instead of hash of actual result.
2nd one triggers at replica 0 and occurs when client 1 sends its 2th request and injects failure of changing operation to get(x) which results in its order statement and result statement using this operation instead of the actual one.
(2) configuration file - config_change_operation_change_result_failure.txt
(3) py -m da -n MainNode --logfile --logfilename logs\log_config_change_operation_change_result_failure_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_change_operation_change_result_failure.txt
py -m da -n ClientNode --logfile --logfilename logs\log_config_change_operation_change_result_failure_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_change_operation_change_result_failure.txt
(4) log files -
logs\log_config_change_operation_change_result_failure_clientnode.txt
logs\log_config_change_operation_change_result_failure_mainnode.txt
(5) outcome - reconfiguration requests will be send for all the failures.
failures[0,1] = result_shuttle(0,5),change_result()
failures[0,0] = client_request(1,1),change_operation()
Test Case 7:
(1) 4 clients having mixed workload of pseudorandom and specific requests. There are 3 failures injected -
1st one triggers at replica 0 and occurs when client 3 sends its 2nd request and injects failure of changing operation to get(x) which results in its order statement and result statement using this operation instead of the actual one.
2nd one is of the case when 1st forwarded request is received at replica 0 but this situation never occurs in phase 2 because reconfiguration is not supported yet.
3rd one triggers at replica 3 and occurs when client 0 sends its 2nd request and injects failure of dropping result statement from the shuttle.
(2) configuration file - config_multiple_clients_multiple_failures.txt
(3) py -m da -n MainNode --logfile --logfilename logs\log_config_multiple_clients_multiple_failures_mainnode.txt -F info --message-buffer-size 100000 src\bcr.da config_multiple_clients_multiple_failures.txt
py -m da -n ClientNode --logfile --logfilename logs\log_config_multiple_clients_multiple_failures_clientnode.txt -F info --message-buffer-size 100000 src\bcr.da config_multiple_clients_multiple_failures.txt
(4) log files -
logs\log_config_multiple_clients_multiple_failures_clientnode.txt
logs\log_config_multiple_clients_multiple_failures_mainnode.txt
(5) outcome - reconfiguration requests will be send for 1st and 3rd failure scenarios.
failures[0,0] = client_request(3,1),change_operation(); forwarded_request(0,0),change_operation()
failures[0,3] = shuttle(0,1),drop_result_stmt()
==============================================================================================================================================================================
PHASE 3
Test Case 1: multiple_clients_checkpt
(1) replicas = 3, clients = 3, total requests among all clients = 16, checkpt_interval = 5, no failures
(2) configuration file - multiple_clients_checkpt.txt
(3) py -m da --logfile --logfilename logs\multiple_clients_checkpt.txt -F info --message-buffer-size 100000 src\bcr.da multiple_clients_checkpt.txt
(4) log files
logs\multiple_clients_checkpt.txt
(5) This is a no failure scenario, purely for testing checkpointing functionality. Total no of checkpoints taken are 3.
Dictionary content at the end is verified and as expected, it doesn't match since there are multiple clients with different requests that other clients don't know.
(6) outcome- PASS
Test Case 2: cpt_ressig_newcon_ordsig_client_sends_reconfig
(1) replicas = 5, clients = 1, total requests among all clients = 10, checkpt_interval = 5, invalid result and order sigs, client sends reconfiguration request
(2) configuration file - cpt_ressig_newcon_ordsig_client_sends_reconfig.txt
(3) py -m da --logfile --logfilename logs\cpt_ressig_newcon_ordsig_client_sends_reconfig.txt -F info --message-buffer-size 100000 src\bcr.da cpt_ressig_newcon_ordsig_client_sends_reconfig.txt
(4) log files
logs\cpt_ressig_newcon_ordsig_client_sends_reconfig.txt
(5) Replica 1 puts invalid result signature leading to client detecting it. Client sends reconfig request to Olympus.
New configuration is formed and order statement is incorrectly signed in this config leading to next replica detecting it and another reconfiguration.
Dictionary content at the end is verified and matches with clients.
(6) outcome- PASS
Test Case 3: multiple_failures
(1) replicas = 5, clients = 1, total requests among all clients = 20, checkpt_interval = 4, multiple failures leading to couple of reconfigurations.
(2) configuration file - multiple_failures.txt
(3) py -m da --logfile --logfilename logs\multiple_failures.txt -F info --message-buffer-size 100000 src\bcr.da multiple_failures.txt
(4) log files
logs\multiple_failures.txt
(5) Change operation is injected which is detected by next replica leading to reconfiguration. On receiving wedge request, replica2 truncates its history.
In new configuration, sleep is injected leading to client's timer timing out and retransmission of request. On receiving checkpoint proof, increment slot is injected.
This leads to hole getting detected on next client request and reconfiguration. Drop on get running state is probabilistic since olympus asks only replica for it's running state.
Dictionary content at the end is verified and matches with clients.
(6) outcome- PASS
Test Case 4: multiple_failures_2
(1) replicas = 3, clients = 1, total requests among all clients = 20, checkpt_interval = 4, multiple failures leading to three reconfigurations.
(2) configuration file - multiple_failures_2.txt
(3) py -m da --logfile --logfilename logs\multiple_failures_2.txt -F info --message-buffer-size 100000 src\bcr.da multiple_failures_2.txt
(4) log files
logs\multiple_failures_2.txt
(5) Change operation is injected which is detected by next replica leading to reconfiguration. On receiving wedge request, replica2 truncates its history.
In new configuration, sleep is injected leading to client's timer timing out and retransmission of request. On receiving checkpoint proof, increment slot is injected.
This leads to hole getting detected on next client request and reconfiguration. Drop on get running state is probabilistic since olympus asks only replica for it's running state.
Dictionary content at the end is verified and matches with clients.
(6) outcome- PASS
Test Case 5: multiple_failures_3
(1) replicas = 3, clients = 1, total requests among all clients = 20, checkpt_interval = 4, multiple failures leading to one reconfiguration.
(2) configuration file - multiple_failures_3.txt
(3) py -m da --logfile --logfilename logs\multiple_failures_3.txt -F info --message-buffer-size 100000 src\bcr.da multiple_failures_3.txt
(4) log files
logs\multiple_failures_3.txt
(5) Extra op is applied to running state, on receiving shuttle for client's 2nd request, replica drops result statement leading to reconfiguration. On catch up, we have sleep for one of the replicas.
Dictionary content at the end is verified and matches with clients.
(6) outcome- PASS
Test Case 6: multihost_head_crash
(1) replicas = 3, clients = 1, total requests among all clients = 5, checkpt_interval = 4, multiple failures leading to one reconfiguration.
(2) configuration file - multihost_head_crash.txt
(3) python3 -m da --logfile --logfilename=logs/multihost_head_crash_mainnode.txt --logfilelevel=info --message-buffer-size=100000 -n MainNode -H 172.17.0.2 -R 172.17.0.3 src/bcr.da multihost_head_crash.txt
python3 -m da --logfile --logfilename=logs/multihost_head_crash_headnode.txt --logfilelevel=info --message-buffer-size=100000 -n HeadNode -D -H 172.17.0.3 -R 172.17.0.2 src/bcr.da multihost_head_crash.txt
(4) log files
logs/multihost_head_crash_mainnode.txt
logs/multihost_head_crash_headnode.txt
(5) Head is running on separate docker container. This is a simple test case where we crash head replica. Reconfiguration is triggered and system resumes to take client requests.
(6) outcome- PASS
=================================================================================================================================================================================
PERFORMANCE EVALUATION
Raft2 single host - 12.93s
BCR perform900.txt single host - 19.87s
BCR perform900.txt on 2 docker containers - 27.10s
Running BCR perform900.txt on docker containers
python3 -m da --logfile --logfilename=logs/mainnodelog.txt --logfilelevel=critical -L critical --message-buffer-size=100000 -n MainNode -H 172.17.0.2 -R 172.17.0.3 src/bcr.da perform900.txt
python3 -m da --logfile --logfilename=logs/headnodelog.txt --logfilelevel=critical -L critical --message-buffer-size=100000 -n HeadNode -D -H 172.17.0.3 -R 172.17.0.2 src/bcr.da perform900.txt
================================================================================================================================================================================
py -m da --logfile --logfilename logs\multiple_clients_checkpt_test.txt -F info --message-buffer-size 100000 -n MainNode src\bcr.da multiple_clients_checkpt.txt