Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JDSMissingTransactions error after a ProvideMissingTransaction sent by JDC #860

Open
GitGab19 opened this issue Apr 18, 2024 · 8 comments · May be fixed by #900
Open

JDSMissingTransactions error after a ProvideMissingTransaction sent by JDC #860

GitGab19 opened this issue Apr 18, 2024 · 8 comments · May be fixed by #900
Assignees
Milestone

Comments

@GitGab19
Copy link
Collaborator

Here's JDS logs:

2024-04-18T14:37:11.133689Z DEBUG roles_logic_sv2::handlers::job_declaration: ProvideMissingTransactionsSuccess: ProvideMissingTransactionsSuccess { request_id: 1234, transaction_list: Seq064K([Ref([2, 0, 0, 0, 0, 1, 1, 123, 74, 100, 196, 183, 60, 123, 5, 125, 194, 119, 156, 14, 204, 131, 183, 1, 105, 196, 195, 167, 140, 40, 226, 22, 77, 245, 249, 93, 223, 87, 211, 0, 0, 0, 0, 0, 253, 255, 255, 255, 2, 105, 77, 0, 0, 0, 0, 0, 0, 22, 0, 20, 239, 223, 0, 61, 228, 159, 222, 79, 2, 164, 143, 49, 241, 237, 182, 215, 196, 142, 95, 145, 188, 190, 235, 242, 0, 0, 0, 0, 22, 0, 20, 51, 39, 10, 0, 89, 23, 124, 152, 47, 212, 184, 113, 107, 14, 114, 147, 129, 47, 239, 116, 1, 64, 95, 135, 181, 135, 99, 101, 148, 42, 169, 125, 114, 169, 13, 139, 79, 231, 60, 136, 6, 199, 12, 82, 158, 210, 61, 135, 210, 21, 122, 55, 138, 180, 76, 40, 114, 98, 147, 138, 235, 45, 106, 152, 92, 95, 63, 184, 57, 74, 128, 63, 193, 199, 175, 194, 48, 246, 137, 209, 215, 205, 113, 33, 207, 192, 202, 122, 39, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 249, 194, 215, 188, 240, 211, 53, 112, 209, 146, 77, 146, 83, 57, 86, 93, 248, 218, 186, 29, 216, 82, 145, 187, 58, 11, 225, 49, 153, 127, 44, 176, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 133, 59, 0, 0, 0, 0, 0, 0, 34, 81, 32, 173, 170, 17, 7, 96, 116, 222, 40, 59, 183, 9, 118, 116, 235, 198, 153, 68, 227, 20, 214, 35, 118, 113, 30, 201, 195, 48, 36, 166, 148, 205, 82, 230, 255, 145, 0, 0, 0, 0, 0, 34, 81, 32, 115, 236, 66, 122, 187, 145, 46, 92, 143, 63, 252, 233, 78, 62, 53, 12, 65, 250, 207, 4, 171, 190, 175, 60, 227, 159, 76, 44, 179, 185, 92, 210, 1, 64, 117, 153, 1, 40, 111, 197, 144, 105, 151, 253, 197, 115, 194, 172, 147, 114, 39, 28, 187, 111, 238, 103, 91, 151, 242, 31, 36, 191, 85, 177, 75, 222, 34, 182, 91, 204, 177, 8, 186, 145, 123, 99, 57, 185, 9, 55, 231, 50, 222, 216, 251, 197, 10, 160, 208, 193, 236, 139, 249, 190, 214, 201, 173, 226, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 203, 16, 60, 183, 114, 86, 45, 238, 6, 115, 160, 91, 219, 223, 167, 127, 187, 14, 46, 32, 153, 106, 121, 151, 162, 154, 70, 159, 170, 76, 137, 82, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 197, 52, 0, 0, 0, 0, 0, 0, 34, 81, 32, 245, 138, 29, 232, 231, 201, 66, 173, 212, 64, 178, 123, 142, 10, 124, 132, 169, 238, 43, 119, 81, 236, 25, 41, 173, 67, 188, 70, 170, 96, 250, 119, 96, 242, 141, 0, 0, 0, 0, 0, 34, 81, 32, 147, 159, 32, 17, 243, 158, 5, 67, 248, 126, 1, 178, 136, 31, 88, 111, 251, 81, 253, 19, 119, 176, 192, 69, 179, 212, 35, 141, 21, 60, 92, 114, 1, 64, 7, 207, 215, 141, 15, 210, 67, 111, 89, 200, 145, 52, 218, 111, 120, 46, 199, 142, 185, 191, 85, 218, 202, 49, 232, 68, 19, 48, 181, 40, 120, 254, 165, 208, 0, 165, 29, 144, 98, 210, 234, 228, 6, 134, 226, 219, 177, 26, 32, 9, 56, 42, 135, 136, 10, 149, 34, 188, 135, 84, 43, 45, 119, 201, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 151, 168, 102, 77, 213, 96, 219, 184, 43, 223, 154, 239, 250, 23, 146, 221, 2, 63, 120, 89, 254, 238, 30, 126, 203, 130, 40, 168, 250, 159, 61, 167, 0, 0, 0, 0, 0, 253, 255, 255, 255, 1, 77, 1, 0, 0, 0, 0, 0, 0, 22, 0, 20, 7, 112, 212, 82, 27, 166, 209, 9, 84, 115, 232, 169, 213, 226, 118, 62, 43, 55, 90, 184, 3, 64, 215, 60, 202, 1, 220, 48, 122, 158, 144, 199, 129, 197, 21, 135, 225, 205, 6, 160, 230, 242, 66, 253, 194, 89, 86, 78, 78, 35, 120, 224, 125, 253, 99, 186, 184, 180, 157, 55, 253, 43, 97, 76, 80, 134, 173, 67, 151, 151, 119, 135, 237, 96, 32, 244, 148, 188, 213, 97, 124, 38, 244, 124, 212, 56, 157, 32, 116, 223, 200, 19, 200, 59, 151, 70, 241, 29, 244, 27, 143, 7, 95, 126, 184, 246, 151, 123, 101, 9, 236, 124, 35, 24, 216, 153, 183, 47, 106, 148, 172, 0, 99, 3, 111, 114, 100, 1, 1, 10, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 1, 7, 49, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 80, 73, 90, 90, 65, 45, 77, 79, 84, 79, 38, 97, 61, 49, 38, 98, 61, 50, 101, 45, 56, 0, 49, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 80, 73, 90, 90, 65, 45, 77, 79, 84, 79, 38, 97, 61, 49, 38, 98, 61, 50, 101, 45, 56, 104, 33, 192, 116, 223, 200, 19, 200, 59, 151, 70, 241, 29, 244, 27, 143, 7, 95, 126, 184, 246, 151, 123, 101, 9, 236, 124, 35, 24, 216, 153, 183, 47, 106, 148, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 237, 24, 0, 144, 223, 241, 14, 104, 58, 110, 97, 230, 45, 20, 51, 72, 114, 8, 219, 93, 7, 97, 7, 49, 55, 117, 245, 207, 11, 154, 237, 23, 0, 0, 0, 0, 0, 253, 255, 255, 255, 1, 77, 1, 0, 0, 0, 0, 0, 0, 22, 0, 20, 87, 127, 20, 236, 183, 83, 115, 127, 220, 72, 30, 125, 194, 186, 102, 181, 205, 239, 68, 224, 3, 64, 209, 95, 120, 140, 74, 156, 115, 2, 224, 232, 82, 236, 244, 93, 90, 189, 42, 77, 251, 114, 226, 30, 98, 76, 192, 247, 119, 237, 217, 7, 101, 123, 238, 248, 164, 18, 152, 58, 139, 215, 89, 187, 128, 140, 130, 209, 240, 70, 216, 15, 148, 210, 215, 241, 93, 57, 23, 46, 65, 249, 193, 213, 91, 141, 185, 32, 25, 61, 75, 201, 85, 163, 203, 227, 149, 162, 253, 125, 222, 42, 254, 163, 250, 166, 192, 210, 42, 191, 93, 63, 220, 243, 234, 72, 114, 63, 78, 8, 172, 0, 99, 3, 111, 114, 100, 1, 1, 10, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 1, 7, 63, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 73, 84, 83, 65, 78, 69, 87, 84, 79, 75, 69, 78, 45, 77, 79, 84, 79, 38, 97, 61, 49, 48, 38, 98, 61, 48, 46, 49, 53, 49, 52, 54, 53, 50, 53, 0, 63, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 73, 84, 83, 65, 78, 69, 87, 84, 79, 75, 69, 78, 45, 77, 79, 84, 79, 38, 97, 61, 49, 48, 38, 98, 61, 48, 46, 49, 53, 49, 52, 54, 53, 50, 53, 104, 33, 192, 25, 61, 75, 201, 85, 163, 203, 227, 149, 162, 253, 125, 222, 42, 254, 163, 250, 166, 192, 210, 42, 191, 93, 63, 220, 243, 234, 72, 114, 63, 78, 8, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 245, 189, 235, 22, 108, 244, 58, 6, 2, 67, 85, 106, 36, 10, 185, 37, 44, 74, 48, 226, 22, 197, 253, 51, 191, 73, 31, 47, 101, 3, 191, 8, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 85, 44, 0, 0, 0, 0, 0, 0, 34, 81, 32, 195, 224, 6, 227, 163, 40, 250, 111, 235, 224, 236, 198, 163, 254, 231, 230, 106, 235, 246, 145, 94, 204, 92, 112, 170, 135, 148, 42, 169, 238, 178, 88, 68, 190, 96, 0, 0, 0, 0, 0, 34, 81, 32, 226, 231, 225, 89, 50, 25, 124, 102, 251, 221, 131, 202, 234, 64, 132, 137, 188, 97, 211, 3, 9, 74, 226, 98, 88, 50, 26, 225, 130, 63, 201, 127, 1, 64, 2, 122, 19, 157, 17, 114, 150, 19, 231, 134, 240, 207, 171, 224, 45, 157, 244, 11, 151, 32, 23, 124, 245, 146, 227, 186, 153, 54, 60, 210, 205, 176, 156, 183, 109, 233, 247, 240, 165, 240, 69, 138, 172, 229, 3, 86, 214, 9, 23, 65, 75, 2, 30, 244, 174, 36, 236, 22, 91, 105, 242, 220, 6, 118, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 31, 2, 251, 162, 132, 42, 24, 172, 182, 244, 141, 247, 214, 250, 74, 59, 167, 80, 102, 93, 172, 13, 125, 22, 60, 59, 151, 11, 126, 103, 21, 25, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 245, 40, 0, 0, 0, 0, 0, 0, 34, 81, 32, 46, 224, 27, 104, 255, 18, 165, 245, 6, 216, 102, 238, 102, 26, 22, 17, 254, 249, 111, 2, 6, 211, 143, 126, 15, 226, 106, 85, 231, 131, 63, 4, 69, 41, 133, 0, 0, 0, 0, 0, 34, 81, 32, 178, 187, 96, 10, 207, 88, 106, 219, 96, 6, 4, 1, 24, 205, 220, 167, 23, 247, 58, 149, 203, 31, 128, 99, 176, 156, 244, 169, 182, 173, 87, 176, 1, 64, 170, 59, 18, 103, 41, 47, 202, 198, 226, 191, 113, 108, 135, 219, 231, 188, 18, 225, 228, 3, 218, 59, 138, 11, 43, 36, 159, 208, 111, 38, 239, 26, 221, 53, 17, 249, 6, 51, 253, 64, 165, 79, 161, 109, 99, 5, 247, 145, 53, 82, 111, 245, 238, 40, 162, 192, 227, 31, 232, 48, 184, 14, 130, 157, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 71, 106, 69, 61, 212, 200, 209, 243, 124, 131, 42, 67, 70, 123, 168, 206, 27, 33, 94, 114, 201, 171, 28, 111, 21, 36, 175, 175, 95, 30, 167, 120, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 245, 40, 0, 0, 0, 0, 0, 0, 34, 81, 32, 77, 243, 220, 251, 214, 211, 211, 200, 134, 169, 108, 161, 176, 60, 228, 184, 59, 79, 103, 23, 245, 155, 29, 87, 84, 139, 74, 82, 235, 169, 131, 240, 24, 47, 233, 19, 2, 0, 0, 0, 22, 0, 20, 116, 122, 17, 93, 110, 87, 223, 252, 189, 13, 186, 189, 239, 225, 205, 104, 60, 55, 31, 75, 2, 71, 48, 68, 2, 32, 29, 217, 50, 113, 36, 31, 23, 179, 177, 181, 56, 119, 110, 60, 92, 249, 130, 240, 153, 215, 223, 108, 215, 123, 183, 55, 218, 43, 42, 157, 8, 186, 2, 32, 38, 159, 232, 23, 174, 171, 58, 110, 185, 68, 9, 254, 252, 91, 132, 212, 0, 185, 205, 131, 117, 111, 39, 173, 164, 222, 104, 227, 228, 250, 175, 166, 1, 33, 2, 252, 34, 171, 22, 143, 251, 44, 127, 175, 97, 209, 44, 154, 242, 26, 93, 117, 33, 241, 206, 175, 196, 193, 132, 13, 186, 125, 190, 143, 160, 189, 130, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 168, 201, 98, 215, 129, 80, 189, 67, 19, 97, 8, 24, 185, 105, 30, 128, 8, 253, 255, 195, 44, 10, 141, 125, 159, 246, 55, 27, 27, 193, 205, 188, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 245, 40, 0, 0, 0, 0, 0, 0, 34, 81, 32, 64, 2, 147, 198, 130, 200, 148, 85, 37, 0, 95, 106, 28, 182, 184, 50, 212, 143, 1, 168, 224, 157, 221, 124, 66, 19, 198, 227, 239, 237, 184, 117, 114, 233, 205, 10, 0, 0, 0, 0, 22, 0, 20, 181, 174, 215, 171, 148, 135, 40, 82, 154, 197, 83, 140, 78, 108, 142, 170, 169, 59, 192, 95, 2, 71, 48, 68, 2, 32, 2, 249, 48, 27, 155, 155, 116, 203, 12, 204, 196, 16, 10, 255, 145, 109, 8, 220, 162, 32, 57, 157, 237, 157, 40, 239, 221, 24, 2, 111, 41, 122, 2, 32, 53, 56, 33, 130, 182, 210, 147, 166, 107, 117, 111, 9, 242, 54, 203, 45, 67, 224, 90, 122, 115, 191, 174, 235, 233, 212, 92, 247, 17, 144, 105, 213, 1, 33, 2, 165, 63, 178, 239, 161, 222, 160, 188, 246, 177, 244, 35, 125, 146, 182, 216, 101, 56, 92, 233, 1, 3, 61, 175, 23, 64, 132, 65, 112, 227, 107, 102, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 195, 117, 124, 150, 123, 69, 161, 223, 155, 89, 83, 106, 174, 29, 215, 47, 98, 238, 16, 62, 60, 19, 135, 7, 248, 204, 198, 27, 123, 152, 104, 177, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 205, 41, 0, 0, 0, 0, 0, 0, 34, 81, 32, 212, 246, 166, 28, 248, 168, 183, 107, 161, 67, 216, 216, 229, 56, 252, 154, 251, 127, 56, 15, 47, 108, 42, 2, 245, 141, 53, 11, 216, 189, 59, 101, 169, 222, 197, 30, 0, 0, 0, 0, 34, 81, 32, 127, 140, 230, 71, 133, 122, 17, 177, 19, 109, 165, 194, 250, 171, 221, 159, 46, 39, 82, 11, 229, 4, 112, 60, 101, 129, 66, 122, 185, 247, 142, 151, 1, 64, 26, 134, 241, 152, 81, 204, 169, 226, 70, 2, 226, 59, 190, 147, 109, 238, 58, 165, 192, 137, 94, 38, 25, 36, 129, 59, 62, 101, 213, 151, 57, 27, 98, 245, 245, 179, 228, 107, 225, 96, 169, 121, 56, 118, 122, 98, 80, 226, 175, 240, 17, 53, 31, 154, 61, 65, 220, 170, 35, 124, 220, 70, 205, 103, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 64, 226, 173, 84, 238, 229, 84, 22, 73, 195, 120, 123, 171, 54, 43, 242, 171, 174, 182, 37, 28, 180, 97, 181, 216, 147, 214, 89, 95, 87, 70, 159, 0, 0, 0, 0, 0, 253, 255, 255, 255, 1, 77, 1, 0, 0, 0, 0, 0, 0, 34, 81, 32, 127, 140, 230, 71, 133, 122, 17, 177, 19, 109, 165, 194, 250, 171, 221, 159, 46, 39, 82, 11, 229, 4, 112, 60, 101, 129, 66, 122, 185, 247, 142, 151, 3, 64, 106, 32, 107, 144, 243, 93, 33, 227, 249, 136, 226, 132, 232, 163, 216, 113, 245, 140, 213, 122, 131, 242, 114, 125, 48, 131, 55, 77, 233, 154, 60, 162, 80, 11, 18, 74, 79, 234, 235, 87, 124, 149, 251, 79, 60, 34, 126, 53, 186, 131, 106, 231, 171, 220, 243, 75, 174, 219, 30, 212, 126, 13, 15, 200, 193, 32, 93, 74, 241, 105, 77, 159, 136, 159, 252, 150, 104, 104, 134, 205, 55, 18, 59, 72, 207, 161, 112, 52, 233, 84, 143, 84, 172, 12, 206, 242, 145, 95, 172, 0, 99, 3, 111, 114, 100, 1, 1, 10, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 1, 7, 67, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 80, 73, 90, 90, 65, 45, 73, 84, 83, 65, 78, 69, 87, 84, 79, 75, 69, 78, 38, 97, 61, 50, 48, 46, 53, 50, 38, 98, 61, 48, 46, 48, 48, 48, 48, 50, 57, 57, 52, 0, 67, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 80, 73, 90, 90, 65, 45, 73, 84, 83, 65, 78, 69, 87, 84, 79, 75, 69, 78, 38, 97, 61, 50, 48, 46, 53, 50, 38, 98, 61, 48, 46, 48, 48, 48, 48, 50, 57, 57, 52, 104, 33, 192, 93, 74, 241, 105, 77, 159, 136, 159, 252, 150, 104, 104, 134, 205, 55, 18, 59, 72, 207, 161, 112, 52, 233, 84, 143, 84, 172, 12, 206, 242, 145, 95, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 11, 40, 110, 112, 58, 26, 59, 129, 189, 158, 209, 65, 214, 159, 209, 18, 93, 116, 86, 36, 227, 32, 79, 8, 3, 185, 51, 231, 145, 32, 193, 99, 0, 0, 0, 0, 0, 253, 255, 255, 255, 1, 77, 1, 0, 0, 0, 0, 0, 0, 34, 81, 32, 70, 98, 16, 10, 119, 15, 151, 18, 145, 55, 36, 174, 110, 165, 91, 58, 168, 71, 224, 31, 125, 206, 190, 66, 189, 160, 109, 71, 16, 242, 125, 124, 3, 64, 103, 131, 248, 57, 148, 213, 144, 203, 187, 228, 143, 80, 49, 125, 244, 136, 142, 46, 202, 221, 116, 160, 119, 68, 125, 42, 253, 83, 213, 90, 173, 252, 190, 246, 81, 75, 204, 39, 71, 76, 222, 232, 220, 121, 245, 78, 94, 236, 230, 116, 207, 160, 193, 210, 174, 168, 97, 147, 138, 232, 63, 79, 186, 45, 173, 32, 38, 245, 14, 173, 216, 69, 154, 116, 3, 38, 161, 236, 43, 229, 129, 41, 49, 169, 99, 206, 136, 30, 73, 180, 81, 182, 159, 248, 67, 145, 191, 21, 172, 0, 99, 3, 111, 114, 100, 1, 1, 10, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 1, 7, 57, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 77, 79, 84, 79, 45, 68, 79, 71, 77, 79, 77, 38, 97, 61, 49, 48, 38, 98, 61, 51, 46, 57, 51, 50, 54, 55, 52, 55, 53, 0, 57, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 77, 79, 84, 79, 45, 68, 79, 71, 77, 79, 77, 38, 97, 61, 49, 48, 38, 98, 61, 51, 46, 57, 51, 50, 54, 55, 52, 55, 53, 104, 33, 192, 38, 245, 14, 173, 216, 69, 154, 116, 3, 38, 161, 236, 43, 229, 129, 41, 49, 169, 99, 206, 136, 30, 73, 180, 81, 182, 159, 248, 67, 145, 191, 21, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 170, 158, 88, 214, 72, 138, 87, 215, 228, 215, 230, 143, 156, 26, 17, 196, 39, 6, 47, 159, 242, 98, 111, 169, 181, 102, 126, 167, 150, 216, 195, 241, 0, 0, 0, 0, 0, 253, 255, 255, 255, 1, 77, 1, 0, 0, 0, 0, 0, 0, 34, 81, 32, 155, 221, 11, 223, 148, 81, 25, 59, 210, 5, 72, 11, 202, 168, 126, 39, 205, 213, 11, 173, 98, 156, 169, 103, 194, 2, 194, 187, 120, 67, 24, 40, 3, 64, 71, 97, 30, 28, 227, 26, 240, 74, 74, 55, 57, 218, 230, 161, 166, 139, 27, 123, 165, 83, 234, 173, 240, 239, 210, 233, 69, 42, 247, 184, 232, 146, 219, 134, 87, 229, 252, 76, 118, 200, 26, 109, 142, 253, 157, 17, 195, 44, 65, 125, 228, 158, 148, 128, 17, 224, 53, 223, 225, 89, 28, 217, 108, 124, 185, 32, 119, 131, 87, 55, 44, 226, 185, 1, 202, 3, 11, 208, 39, 159, 185, 87, 153, 3, 180, 237, 81, 169, 10, 170, 235, 141, 154, 152, 230, 234, 147, 165, 172, 0, 99, 3, 111, 114, 100, 1, 1, 10, 116, 101, 120, 116, 47, 112, 108, 97, 105, 110, 1, 7, 63, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 77, 79, 84, 79, 45, 80, 73, 90, 90, 65, 38, 97, 61, 48, 46, 48, 48, 48, 48, 49, 38, 98, 61, 50, 52, 49, 46, 56, 49, 49, 49, 51, 52, 55, 50, 0, 63, 64, 109, 111, 116, 111, 58, 115, 119, 97, 112, 58, 58, 99, 98, 114, 99, 45, 50, 48, 58, 115, 119, 97, 112, 63, 97, 98, 61, 77, 79, 84, 79, 45, 80, 73, 90, 90, 65, 38, 97, 61, 48, 46, 48, 48, 48, 48, 49, 38, 98, 61, 50, 52, 49, 46, 56, 49, 49, 49, 51, 52, 55, 50, 104, 33, 193, 119, 131, 87, 55, 44, 226, 185, 1, 202, 3, 11, 208, 39, 159, 185, 87, 153, 3, 180, 237, 81, 169, 10, 170, 235, 141, 154, 152, 230, 234, 147, 165, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 112, 117, 138, 122, 167, 221, 213, 40, 65, 208, 108, 99, 246, 6, 182, 78, 89, 186, 217, 139, 148, 86, 231, 151, 149, 54, 94, 238, 10, 21, 84, 134, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 5, 46, 0, 0, 0, 0, 0, 0, 34, 81, 32, 144, 136, 1, 240, 193, 78, 146, 61, 217, 4, 9, 152, 187, 7, 106, 122, 207, 43, 170, 102, 157, 127, 106, 175, 76, 57, 222, 44, 111, 98, 11, 207, 36, 250, 1, 0, 0, 0, 0, 0, 22, 0, 20, 3, 217, 253, 60, 54, 177, 159, 154, 122, 252, 166, 0, 236, 138, 125, 243, 59, 248, 124, 228, 2, 71, 48, 68, 2, 32, 48, 134, 94, 139, 54, 192, 159, 29, 150, 202, 223, 185, 30, 56, 97, 140, 162, 159, 198, 85, 31, 172, 177, 19, 192, 52, 118, 38, 93, 97, 225, 227, 2, 32, 33, 11, 215, 127, 47, 42, 101, 63, 40, 180, 44, 87, 153, 253, 236, 185, 22, 101, 40, 179, 13, 2, 73, 108, 235, 151, 31, 252, 202, 49, 50, 255, 1, 33, 2, 42, 233, 92, 162, 221, 54, 29, 86, 251, 144, 187, 4, 42, 203, 86, 246, 88, 189, 215, 120, 176, 252, 27, 31, 82, 208, 34, 170, 98, 159, 72, 252, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 235, 16, 115, 84, 56, 75, 190, 227, 103, 178, 114, 52, 175, 86, 182, 148, 209, 162, 221, 56, 251, 242, 233, 118, 112, 93, 201, 117, 185, 230, 237, 104, 2, 0, 0, 0, 0, 255, 255, 255, 255, 2, 128, 150, 152, 0, 0, 0, 0, 0, 22, 0, 20, 12, 78, 96, 30, 159, 125, 178, 89, 143, 191, 233, 195, 228, 160, 112, 4, 140, 80, 91, 201, 107, 159, 165, 158, 0, 0, 0, 0, 34, 81, 32, 208, 153, 49, 135, 163, 152, 153, 96, 46, 247, 101, 100, 25, 63, 242, 221, 164, 180, 117, 196, 2, 45, 195, 230, 142, 205, 125, 165, 92, 197, 102, 149, 1, 64, 159, 32, 64, 255, 171, 244, 107, 220, 147, 40, 124, 149, 36, 41, 235, 213, 112, 17, 235, 75, 159, 152, 118, 156, 132, 235, 130, 201, 106, 158, 104, 230, 243, 115, 236, 46, 165, 16, 100, 247, 81, 223, 6, 52, 213, 85, 104, 52, 191, 80, 155, 220, 113, 51, 20, 2, 70, 243, 217, 196, 155, 140, 49, 153, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 153, 17, 180, 132, 12, 83, 100, 16, 176, 9, 22, 95, 54, 239, 21, 66, 251, 145, 173, 156, 117, 78, 0, 199, 54, 218, 90, 96, 115, 23, 128, 9, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 245, 40, 0, 0, 0, 0, 0, 0, 34, 81, 32, 179, 36, 97, 242, 22, 154, 178, 206, 97, 211, 243, 31, 216, 122, 68, 174, 126, 140, 125, 226, 32, 130, 55, 237, 154, 1, 231, 198, 1, 119, 91, 93, 138, 7, 52, 2, 0, 0, 0, 0, 34, 81, 32, 191, 49, 118, 121, 186, 218, 93, 204, 186, 237, 189, 237, 72, 59, 179, 190, 37, 103, 56, 13, 59, 168, 174, 37, 13, 246, 218, 12, 81, 67, 242, 150, 1, 64, 179, 205, 146, 120, 238, 206, 176, 187, 224, 217, 14, 28, 85, 201, 222, 65, 236, 18, 218, 6, 121, 57, 88, 114, 86, 218, 228, 189, 10, 21, 166, 28, 70, 210, 88, 75, 54, 219, 212, 163, 89, 108, 220, 65, 62, 99, 76, 144, 82, 15, 114, 208, 22, 253, 168, 244, 72, 175, 139, 77, 80, 22, 56, 177, 0, 0, 0, 0]), Ref([2, 0, 0, 0, 0, 1, 1, 143, 87, 89, 95, 162, 107, 17, 138, 236, 107, 122, 214, 61, 91, 54, 121, 20, 145, 112, 79, 58, 181, 181, 181, 245, 235, 20, 192, 74, 32, 186, 122, 1, 0, 0, 0, 0, 255, 255, 255, 255, 2, 16, 39, 0, 0, 0, 0, 0, 0, 22, 0, 20, 153, 30, 18, 127, 231, 6, 73, 72, 68, 228, 67, 44, 34, 75, 106, 159, 1, 91, 133, 251, 134, 237, 0, 0, 0, 0, 0, 0, 34, 81, 32, 194, 57, 196, 88, 4, 12, 180, 105, 241, 203, 94, 211, 44, 41, 133, 116, 95, 3, 189, 23, 156, 9, 106, 112, 169, 163, 154, 92, 157, 47, 117, 62, 1, 64, 198, 95, 224, 221, 215, 210, 26, 40, 190, 46, 138, 118, 77, 34, 248, 120, 111, 58, 213, 221, 76, 111, 72, 38, 62, 169, 121, 68, 224, 245, 206, 132, 118, 255, 222, 216, 46, 127, 158, 129, 28, 53, 24, 188, 83, 124, 50, 37, 196, 12, 76, 66, 99, 90, 59, 113, 24, 149, 92, 109, 102, 121, 245, 96, 0, 0, 0, 0])], PhantomData<&binary_codec_sv2::datatypes::non_copy_data_types::inner::Inner<false, 1, 3, 16777215>>) }
2024-04-18T14:37:11.134932Z ERROR jd_server::lib::job_declarator: JDSMissingTransactions
2024-04-18T14:37:11.134936Z DEBUG jd_server::lib::status: Error: Custom("Invalid message received")
2024-04-18T14:37:11.134957Z  INFO jd_server: HEALTHY message: Custom SV2 error: `"Invalid message received"`
2024-04-18T14:37:11.134979Z ERROR network_helpers_sv2::noise_connection_tokio: Disconnecting from client due to error receiving: receiving from an empty and closed channel - 127.0.0.1:52522

And JDC logs:

2024-04-18T14:37:10.004975Z  INFO roles_logic_sv2::handlers::template_distribution: Received SetNewPrevHash for template: 14737
2024-04-18T14:37:10.004980Z DEBUG roles_logic_sv2::handlers::template_distribution: SetNewPrevHash: SetNewPrevHash { template_id: 14737, prev_hash: Ref([126, 39, 101, 7, 167, 211, 112, 42, 73, 145, 164, 78, 238, 87, 36, 90, 61, 208, 66, 1, 113, 8, 240, 11, 43, 0, 0, 0, 0, 0, 0, 0]), header_timestamp: 1713452169, n_bits: 422451157, target: Ref([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 213, 23, 46, 0, 0, 0, 0, 0, 0, 0]) }
2024-04-18T14:37:10.004989Z  INFO jd_client::lib::template_receiver: Received SetNewPrevHash, waiting for IS_NEW_TEMPLATE_HANDLED
2024-04-18T14:37:10.004994Z  INFO jd_client::lib::template_receiver: IS_NEW_TEMPLATE_HANDLED ok
2024-04-18T14:37:10.005226Z  INFO roles_logic_sv2::handlers::mining: Received new extended mining job for channel id: 5 with job id: 5848 is_future: true
2024-04-18T14:37:10.005235Z DEBUG roles_logic_sv2::handlers::mining: NewExtendedMiningJob: NewExtendedMiningJob { channel_id: 5, job_id: 5848, min_ntime: Sv2Option([], PhantomData<&u32>), version: 536870912, version_rolling_allowed: true, merkle_path: Seq0255([Ref([51, 51, 38, 25, 157, 161, 183, 180, 194, 212, 86, 209, 85, 16, 194, 208, 32, 30, 89, 177, 105, 167, 189, 36, 205, 229, 89, 18, 64, 154, 221, 206]), Ref([196, 75, 173, 2, 232, 73, 12, 172, 241, 18, 131, 76, 19, 124, 76, 9, 71, 253, 36, 171, 107, 110, 163, 91, 145, 247, 37, 165, 162, 162, 240, 73]), Ref([23, 121, 173, 66, 171, 91, 178, 105, 240, 10, 40, 184, 203, 43, 112, 203, 206, 65, 46, 167, 79, 105, 104, 71, 90, 73, 78, 130, 122, 151, 32, 120]), Ref([221, 207, 8, 170, 245, 176, 70, 110, 79, 5, 21, 57, 190, 184, 193, 218, 37, 93, 216, 84, 65, 248, 21, 103, 158, 62, 41, 149, 162, 193, 45, 157]), Ref([92, 12, 64, 216, 169, 176, 255, 220, 198, 207, 134, 205, 110, 142, 140, 162, 184, 243, 120, 195, 61, 191, 211, 38, 167, 32, 189, 16, 252, 148, 164, 52]), Ref([5, 223, 248, 50, 25, 25, 12, 96, 74, 69, 124, 202, 90, 86, 42, 97, 119, 44, 21, 194, 49, 122, 175, 33, 162, 230, 210, 195, 44, 205, 194, 155]), Ref([0, 83, 176, 175, 181, 109, 62, 59, 130, 234, 25, 161, 161, 171, 51, 15, 199, 99, 41, 105, 193, 85, 61, 58, 163, 87, 121, 153, 99, 95, 131, 213]), Ref([47, 3, 169, 92, 98, 138, 38, 204, 116, 118, 81, 230, 119, 212, 230, 172, 78, 147, 232, 66, 187, 213, 231, 52, 239, 74, 177, 119, 198, 37, 38, 242]), Ref([31, 177, 175, 201, 253, 99, 35, 85, 138, 229, 38, 169, 183, 125, 55, 230, 161, 183, 213, 173, 100, 166, 50, 179, 230, 55, 218, 144, 242, 207, 209, 39])], PhantomData<&binary_codec_sv2::datatypes::non_copy_data_types::inner::Inner<true, 32, 0, 0>>), coinbase_tx_prefix: Ref([2, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 56, 3, 204, 122, 39, 0, 83, 116, 114, 97, 116, 117, 109, 32, 118, 50, 32, 83, 82, 73, 32, 80, 111, 111, 108]), coinbase_tx_suffix: Ref([255, 255, 255, 255, 2, 142, 183, 204, 2, 0, 0, 0, 0, 22, 0, 20, 235, 225, 183, 220, 194, 147, 204, 170, 14, 231, 67, 168, 111, 137, 223, 130, 88, 194, 8, 252, 0, 0, 0, 0, 0, 0, 0, 0, 38, 106, 36, 170, 33, 169, 237, 238, 193, 214, 27, 77, 84, 9, 186, 85, 180, 163, 40, 117, 194, 87, 32, 21, 208, 179, 223, 178, 163, 210, 218, 30, 226, 111, 152, 205, 200, 85, 217, 1, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) }
2024-04-18T14:37:10.005275Z  WARN jd_client::lib::upstream_sv2::upstream: Extended job received from upstream, proxy ignore it, and use the one declared by JOB DECLARATOR
2024-04-18T14:37:10.005297Z  INFO roles_logic_sv2::handlers::mining: Received SetNewPrevHash channel id: 5, job id: 5848
2024-04-18T14:37:10.005310Z DEBUG roles_logic_sv2::handlers::mining: SetNewPrevHash: SetNewPrevHash { channel_id: 5, job_id: 5848, prev_hash: Ref([126, 39, 101, 7, 167, 211, 112, 42, 73, 145, 164, 78, 238, 87, 36, 90, 61, 208, 66, 1, 113, 8, 240, 11, 43, 0, 0, 0, 0, 0, 0, 0]), min_ntime: 1713452169, nbits: 422451157 }
2024-04-18T14:37:10.005322Z  WARN jd_client::lib::upstream_sv2::upstream: SNPH received from upstream, proxy ignore it, and use the one declared by JOB DECLARATOR
2024-04-18T14:37:10.191249Z  INFO roles_logic_sv2::handlers::template_distribution: Received NewTemplate with id: 14739, is future: false
2024-04-18T14:37:10.191264Z DEBUG roles_logic_sv2::handlers::template_distribution: NewTemplate: NewTemplate { template_id: 14739, future_template: false, version: 536870912, coinbase_tx_version: 2, coinbase_prefix: Ref([3, 204, 122, 39, 0]), coinbase_tx_input_sequence: 4294967295, coinbase_tx_value_remaining: 47003954, coinbase_tx_outputs_count: 1, coinbase_tx_outputs: Ref([0, 0, 0, 0, 0, 0, 0, 0, 38, 106, 36, 170, 33, 169, 237, 205, 142, 167, 113, 41, 125, 77, 86, 122, 64, 255, 109, 19, 78, 41, 31, 160, 98, 160, 201, 152, 205, 245, 33, 175, 239, 201, 248, 60, 69, 32, 155]), coinbase_tx_locktime: 0, merkle_path: Seq0255([Ref([51, 51, 38, 25, 157, 161, 183, 180, 194, 212, 86, 209, 85, 16, 194, 208, 32, 30, 89, 177, 105, 167, 189, 36, 205, 229, 89, 18, 64, 154, 221, 206]), Ref([196, 75, 173, 2, 232, 73, 12, 172, 241, 18, 131, 76, 19, 124, 76, 9, 71, 253, 36, 171, 107, 110, 163, 91, 145, 247, 37, 165, 162, 162, 240, 73]), Ref([93, 89, 71, 48, 211, 196, 92, 51, 26, 221, 123, 75, 47, 134, 189, 51, 224, 30, 242, 109, 220, 7, 183, 74, 141, 103, 140, 26, 7, 97, 151, 103]), Ref([83, 239, 70, 108, 38, 92, 245, 43, 198, 193, 81, 47, 32, 59, 255, 178, 56, 211, 120, 197, 194, 61, 203, 53, 20, 244, 227, 238, 218, 168, 156, 181]), Ref([76, 58, 54, 142, 31, 230, 226, 216, 126, 44, 168, 161, 205, 113, 2, 211, 231, 39, 0, 147, 81, 245, 38, 194, 213, 85, 94, 11, 253, 202, 236, 19]), Ref([122, 186, 99, 35, 102, 83, 16, 81, 162, 84, 207, 246, 19, 84, 233, 243, 164, 91, 17, 168, 238, 75, 161, 225, 213, 121, 211, 50, 8, 145, 7, 93]), Ref([205, 16, 19, 105, 144, 120, 166, 246, 134, 111, 94, 35, 175, 172, 146, 194, 71, 105, 90, 115, 254, 8, 18, 172, 126, 246, 58, 201, 24, 176, 82, 132]), Ref([122, 215, 206, 23, 163, 235, 160, 47, 102, 5, 6, 150, 144, 141, 97, 211, 234, 195, 208, 47, 208, 95, 201, 233, 80, 11, 213, 190, 177, 141, 75, 90]), Ref([183, 22, 8, 235, 126, 218, 116, 166, 168, 77, 187, 251, 42, 141, 26, 121, 154, 248, 90, 34, 248, 128, 249, 164, 113, 30, 226, 140, 101, 7, 118, 244])], PhantomData<&binary_codec_sv2::datatypes::non_copy_data_types::inner::Inner<true, 32, 0, 0>>) }
2024-04-18T14:37:10.191331Z DEBUG roles_logic_sv2::job_creator: New extended mining job created: NewExtendedMiningJob { channel_id: 0, job_id: 1236, min_ntime: Sv2Option([1713451030], PhantomData<&u32>), version: 536870912, version_rolling_allowed: true, merkle_path: Seq0255([Owned([51, 51, 38, 25, 157, 161, 183, 180, 194, 212, 86, 209, 85, 16, 194, 208, 32, 30, 89, 177, 105, 167, 189, 36, 205, 229, 89, 18, 64, 154, 221, 206]), Owned([196, 75, 173, 2, 232, 73, 12, 172, 241, 18, 131, 76, 19, 124, 76, 9, 71, 253, 36, 171, 107, 110, 163, 91, 145, 247, 37, 165, 162, 162, 240, 73]), Owned([93, 89, 71, 48, 211, 196, 92, 51, 26, 221, 123, 75, 47, 134, 189, 51, 224, 30, 242, 109, 220, 7, 183, 74, 141, 103, 140, 26, 7, 97, 151, 103]), Owned([83, 239, 70, 108, 38, 92, 245, 43, 198, 193, 81, 47, 32, 59, 255, 178, 56, 211, 120, 197, 194, 61, 203, 53, 20, 244, 227, 238, 218, 168, 156, 181]), Owned([76, 58, 54, 142, 31, 230, 226, 216, 126, 44, 168, 161, 205, 113, 2, 211, 231, 39, 0, 147, 81, 245, 38, 194, 213, 85, 94, 11, 253, 202, 236, 19]), Owned([122, 186, 99, 35, 102, 83, 16, 81, 162, 84, 207, 246, 19, 84, 233, 243, 164, 91, 17, 168, 238, 75, 161, 225, 213, 121, 211, 50, 8, 145, 7, 93]), Owned([205, 16, 19, 105, 144, 120, 166, 246, 134, 111, 94, 35, 175, 172, 146, 194, 71, 105, 90, 115, 254, 8, 18, 172, 126, 246, 58, 201, 24, 176, 82, 132]), Owned([122, 215, 206, 23, 163, 235, 160, 47, 102, 5, 6, 150, 144, 141, 97, 211, 234, 195, 208, 47, 208, 95, 201, 233, 80, 11, 213, 190, 177, 141, 75, 90]), Owned([183, 22, 8, 235, 126, 218, 116, 166, 168, 77, 187, 251, 42, 141, 26, 121, 154, 248, 90, 34, 248, 128, 249, 164, 113, 30, 226, 140, 101, 7, 118, 244])], PhantomData<&binary_codec_sv2::datatypes::non_copy_data_types::inner::Inner<true, 32, 0, 0>>), coinbase_tx_prefix: Owned([2, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 56, 3, 204, 122, 39, 0, 83, 116, 114, 97, 116, 117, 109, 32, 118, 50, 32, 83, 82, 73, 32, 80, 111, 111, 108]), coinbase_tx_suffix: Owned([255, 255, 255, 255, 2, 50, 57, 205, 2, 0, 0, 0, 0, 22, 0, 20, 235, 225, 183, 220, 194, 147, 204, 170, 14, 231, 67, 168, 111, 137, 223, 130, 88, 194, 8, 252, 0, 0, 0, 0, 0, 0, 0, 0, 38, 106, 36, 170, 33, 169, 237, 205, 142, 167, 113, 41, 125, 77, 86, 122, 64, 255, 109, 19, 78, 41, 31, 160, 98, 160, 201, 152, 205, 245, 33, 175, 239, 201, 248, 60, 69, 32, 155, 1, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) }
2024-04-18T14:37:10.191620Z  INFO roles_logic_sv2::handlers::mining: Received new extended mining job for channel id: 5 with job id: 5849 is_future: false
2024-04-18T14:37:10.191626Z DEBUG roles_logic_sv2::handlers::mining: NewExtendedMiningJob: NewExtendedMiningJob { channel_id: 5, job_id: 5849, min_ntime: Sv2Option([1713451030], PhantomData<&u32>), version: 536870912, version_rolling_allowed: true, merkle_path: Seq0255([Ref([51, 51, 38, 25, 157, 161, 183, 180, 194, 212, 86, 209, 85, 16, 194, 208, 32, 30, 89, 177, 105, 167, 189, 36, 205, 229, 89, 18, 64, 154, 221, 206]), Ref([196, 75, 173, 2, 232, 73, 12, 172, 241, 18, 131, 76, 19, 124, 76, 9, 71, 253, 36, 171, 107, 110, 163, 91, 145, 247, 37, 165, 162, 162, 240, 73]), Ref([93, 89, 71, 48, 211, 196, 92, 51, 26, 221, 123, 75, 47, 134, 189, 51, 224, 30, 242, 109, 220, 7, 183, 74, 141, 103, 140, 26, 7, 97, 151, 103]), Ref([83, 239, 70, 108, 38, 92, 245, 43, 198, 193, 81, 47, 32, 59, 255, 178, 56, 211, 120, 197, 194, 61, 203, 53, 20, 244, 227, 238, 218, 168, 156, 181]), Ref([76, 58, 54, 142, 31, 230, 226, 216, 126, 44, 168, 161, 205, 113, 2, 211, 231, 39, 0, 147, 81, 245, 38, 194, 213, 85, 94, 11, 253, 202, 236, 19]), Ref([122, 186, 99, 35, 102, 83, 16, 81, 162, 84, 207, 246, 19, 84, 233, 243, 164, 91, 17, 168, 238, 75, 161, 225, 213, 121, 211, 50, 8, 145, 7, 93]), Ref([138, 123, 128, 48, 101, 71, 34, 138, 17, 243, 0, 22, 194, 46, 196, 127, 209, 69, 217, 178, 170, 218, 228, 25, 189, 119, 247, 185, 227, 73, 198, 60]), Ref([153, 253, 118, 236, 34, 39, 92, 169, 72, 29, 62, 65, 184, 173, 31, 119, 111, 146, 215, 208, 73, 109, 67, 114, 129, 11, 96, 229, 217, 111, 52, 2]), Ref([83, 111, 136, 30, 144, 46, 151, 74, 82, 55, 216, 46, 36, 193, 208, 76, 92, 117, 60, 166, 89, 62, 194, 64, 153, 229, 219, 232, 152, 84, 148, 246])], PhantomData<&binary_codec_sv2::datatypes::non_copy_data_types::inner::Inner<true, 32, 0, 0>>), coinbase_tx_prefix: Ref([2, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 255, 56, 3, 204, 122, 39, 0, 83, 116, 114, 97, 116, 117, 109, 32, 118, 50, 32, 83, 82, 73, 32, 80, 111, 111, 108]), coinbase_tx_suffix: Ref([255, 255, 255, 255, 2, 108, 39, 205, 2, 0, 0, 0, 0, 22, 0, 20, 235, 225, 183, 220, 194, 147, 204, 170, 14, 231, 67, 168, 111, 137, 223, 130, 88, 194, 8, 252, 0, 0, 0, 0, 0, 0, 0, 0, 38, 106, 36, 170, 33, 169, 237, 19, 234, 139, 239, 122, 74, 5, 3, 77, 87, 152, 166, 83, 142, 242, 54, 236, 141, 159, 85, 190, 91, 61, 41, 2, 196, 201, 129, 87, 22, 6, 235, 1, 32, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) }
2024-04-18T14:37:10.191660Z  WARN jd_client::lib::upstream_sv2::upstream: Extended job received from upstream, proxy ignore it, and use the one declared by JOB DECLARATOR
2024-04-18T14:37:10.252728Z  INFO roles_logic_sv2::handlers::template_distribution: Received RequestTransactionDataSuccess for template: 14737
2024-04-18T14:37:10.356387Z  INFO roles_logic_sv2::handlers::template_distribution: Received RequestTransactionDataSuccess for template: 14739
2024-04-18T14:37:10.710612Z  INFO roles_logic_sv2::handlers::job_declaration: Received ProvideMissingTransactions with id: 1234
2024-04-18T14:37:10.710623Z DEBUG roles_logic_sv2::handlers::job_declaration: ProvideMissingTransactions: ProvideMissingTransactions { request_id: 1234, unknown_tx_position_list: Seq064K([2, 12, 28, 31, 43, 56, 58, 62, 64, 66, 67, 71, 73, 78, 83, 85, 86], PhantomData<&u16>) }
2024-04-18T14:37:11.133751Z DEBUG roles_logic_sv2::handlers::job_declaration: Received AllocateMiningJobTokenSuccess with id: 0
2024-04-18T14:37:11.133857Z  INFO roles_logic_sv2::handlers::job_declaration: Received ProvideMissingTransactions with id: 1235
2024-04-18T14:37:11.133862Z DEBUG roles_logic_sv2::handlers::job_declaration: ProvideMissingTransactions: ProvideMissingTransactions { request_id: 1235, unknown_tx_position_list: Seq064K([2, 3, 13, 29, 32, 44, 57, 59, 63, 65, 67, 68, 69, 73, 75, 77, 82, 87, 89, 90, 97], PhantomData<&u16>) }
2024-04-18T14:37:11.134037Z DEBUG roles_logic_sv2::handlers::job_declaration: Received AllocateMiningJobTokenSuccess with id: 0
2024-04-18T14:37:11.134042Z DEBUG roles_logic_sv2::handlers::job_declaration: Received AllocateMiningJobTokenSuccess with id: 1
2024-04-18T14:37:11.134991Z ERROR network_helpers_sv2::noise_connection_tokio: Disconnected from client while reading : early eof - 127.0.0.1:34264
thread 'tokio-runtime-worker' panicked at jd-client/src/lib/job_declarator/mod.rs:263:72:
called `Result::unwrap()` on an `Err` value: RecvError
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
@Fi3
Copy link
Collaborator

Fi3 commented Apr 24, 2024

in the jd_client the handler for the ProvideMissingTransactionData message silently ignore invalid request:

    fn handle_provide_missing_transactions(
        &mut self,
        message: ProvideMissingTransactions,
    ) -> Result<SendTo, Error> {
        let tx_list = self
            .last_declare_mining_job_sent
            .clone()
            .unwrap()
            .tx_list
            .into_inner();
        let unknown_tx_position_list: Vec<u16> = message.unknown_tx_position_list.into_inner();
        let missing_transactions: Vec<binary_sv2::B016M> = unknown_tx_position_list
            .iter()
            .filter_map(|&pos| tx_list.get(pos as usize).cloned())
            .collect();
        let request_id = message.request_id;
        let message_provide_missing_transactions = ProvideMissingTransactionsSuccess {
            request_id,
            transaction_list: binary_sv2::Seq064K::new(missing_transactions).unwrap(),
        };
        let message_enum =
            JobDeclaration::ProvideMissingTransactionsSuccess(message_provide_missing_transactions);
        Ok(SendTo::Respond(message_enum))
    }

The issue here is filter_map that when the requested data is not present instead returning and error it just ignore the requested data and do not add it the the returned message. This could lead to inconsistencies between the upstream and the downstream state. And in particular to the JDSMissingTransactions error.

Also if this should be fixed, this is not the cause at least sometimes is not. Running the client + server for several hours with the below patch:

        if missing_transactions.len() != unknown_tx_position_list.len() {
           std::process::abort();
       };

We still have the JDSMissingTransactions error without having a jdc failure. (that means that the jdc send all the requested data upstream but upstream still think that some txs is missing)

@Fi3
Copy link
Collaborator

Fi3 commented Apr 24, 2024

Looking at the code my best guess is that something like the below happens:

Client Server MissingTxs server state
declare1
handle_declare set for declare1
provide_miss_txs_1
declare2
handle_declare set for declare2
provide_miss_txs_2
handle_provide_miss
prov_miss_txs_succ_1
handle_provide_miss_succ

In that case when the jds call handle_provide_miss_succ with prov_miss_txs_succ_1 the downstream state is already set to the second declare message.

We should leverage the MG to test things like that. MG tests are thought to be highly composable. So we could have a file with several valid bitcoin txs serialized, that can be used in the tests. For example here we should add a test for the JDS, the test will act like a JDC and it will look somthing like that:

  1. spawn jds that point to hosted tp
  2. connect to jds and send decalre mining job (we will contruct the message using the file with all the serialized txs so that we wont have all that crap on the actual test, also this txs can not be in mempool)
  3. check that jds send ProvideMissingTransaction, ignore it
  4. send another declare with different txs set
  5. check that jds send ProvideMissingTransaction, ignore it
  6. send ProvideMissingTransactionSucc for the first ProvideMissingTransaction received

@Fi3
Copy link
Collaborator

Fi3 commented Apr 24, 2024

In order to fix it, I think that we could add a ring buffer with capacity 3 (should be enough) where we put the request ids of the ProvideMissingTransactions. When we receive a ProvideMissingTransactionsSuccess we check the id,

  • if is the last id in the ring buffer we do what we are doing now
  • if is in the ring buffer but is not the last (we check for that after having checked if is the last, if make sense we could also save the last somewhere else so that we don't have to check the ring buffer 99% of the times). In that case we just ignore the message.
  • else we return an error

@Fi3
Copy link
Collaborator

Fi3 commented Apr 24, 2024

even simpler would be correct also ingore every ProvideMissingTransactionsSuccess that have the wrong id

@pavlenex pavlenex added this to the Milestone 5 milestone Apr 29, 2024
@pavlenex pavlenex added the spec incompatibility Issues ensuring SRI follows the specs label May 1, 2024
@pavlenex pavlenex mentioned this issue May 1, 2024
@pavlenex pavlenex modified the milestones: 1.2.0, 1.0.1 May 1, 2024
@pavlenex pavlenex removed the spec incompatibility Issues ensuring SRI follows the specs label May 2, 2024
@pavlenex pavlenex modified the milestones: 1.0.1, 1.1.0 May 2, 2024
@GitGab19
Copy link
Collaborator Author

GitGab19 commented May 6, 2024

in the jd_client the handler for the ProvideMissingTransactionData message silently ignore invalid request:

    fn handle_provide_missing_transactions(
        &mut self,
        message: ProvideMissingTransactions,
    ) -> Result<SendTo, Error> {
        let tx_list = self
            .last_declare_mining_job_sent
            .clone()
            .unwrap()
            .tx_list
            .into_inner();
        let unknown_tx_position_list: Vec<u16> = message.unknown_tx_position_list.into_inner();
        let missing_transactions: Vec<binary_sv2::B016M> = unknown_tx_position_list
            .iter()
            .filter_map(|&pos| tx_list.get(pos as usize).cloned())
            .collect();
        let request_id = message.request_id;
        let message_provide_missing_transactions = ProvideMissingTransactionsSuccess {
            request_id,
            transaction_list: binary_sv2::Seq064K::new(missing_transactions).unwrap(),
        };
        let message_enum =
            JobDeclaration::ProvideMissingTransactionsSuccess(message_provide_missing_transactions);
        Ok(SendTo::Respond(message_enum))
    }

The issue here is filter_map that when the requested data is not present instead returning and error it just ignore the requested data and do not add it the the returned message. This could lead to inconsistencies between the upstream and the downstream state. And in particular to the JDSMissingTransactions error.

Also if this should be fixed, this is not the cause at least sometimes is not. Running the client + server for several hours with the below patch:

        if missing_transactions.len() != unknown_tx_position_list.len() {
           std::process::abort();
       };

We still have the JDSMissingTransactions error without having a jdc failure. (that means that the jdc send all the requested data upstream but upstream still think that some txs is missing)

I'm running the JDC with this patch, I will let you know.

@GitGab19
Copy link
Collaborator Author

GitGab19 commented May 6, 2024

Looking at the code my best guess is that something like the below happens:
Client Server MissingTxs server state
declare1
handle_declare set for declare1
provide_miss_txs_1
declare2
handle_declare set for declare2
provide_miss_txs_2
handle_provide_miss
prov_miss_txs_succ_1
handle_provide_miss_succ

In that case when the jds call handle_provide_miss_succ with prov_miss_txs_succ_1 the downstream state is already set to the second declare message.

We should leverage the MG to test things like that. MG tests are thought to be highly composable. So we could have a file with several valid bitcoin txs serialized, that can be used in the tests. For example here we should add a test for the JDS, the test will act like a JDC and it will look somthing like that:

1. spawn jds that point to hosted tp

2. connect to jds and send decalre mining job (we will contruct the message using the file with all the serialized txs so that we wont have all that crap on the actual test, also this txs can not be in mempool)

3. check that jds send ProvideMissingTransaction, ignore it

4. send another declare with different txs set

5. check that jds send ProvideMissingTransaction, ignore it

6. send ProvideMissingTransactionSucc for the first ProvideMissingTransaction received

I'm also pretty confident this is exactly what is happening. It seems to me that this can go wrong after new blocks found (not completely sure if it's related to it btw)

@GitGab19
Copy link
Collaborator Author

GitGab19 commented May 6, 2024

even simpler would be correct also ingore every ProvideMissingTransactionsSuccess that have the wrong id

If the problem is the one you described, I would go with this solution

@GitGab19
Copy link
Collaborator Author

GitGab19 commented May 6, 2024

To give more context and confirm your thesis, here's a screenshot of JDC logs:
Screenshot 2024-05-06 at 11 03 20

The issue happens sometimes after a new block is found, if time between two blocks is very little.
I would try to ignore ProvideMissingTransactionsSuccess with wrong (old) ids on JDS side as you suggested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress 🏗️
Development

Successfully merging a pull request may close this issue.

3 participants