/
CHANGE_LOG
4986 lines (4011 loc) · 218 KB
/
CHANGE_LOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
6.15.0 2023-12-06T12:37:33-0700
Build:
* Eliminated all *.hin files (except "registry/registry.hin") and extractDecls(1) in order
to make the development and build process more conventional and easier to understand
* Corrected creation of runtime link to accomodate non-standard installs;
* Makefile.am: Corrected "ensure_ldmhome_links" rule
* ensure_hdmhome_links: Added extraction of subdirectory name from installation prefix
ldmadmin(1):
* "queuecheck" command: Placed the pathname of the product-queue in quotes just in case it
contains embedded whitespace
* "showsettings" command: Added "pqmon -S"
* Modified "vetqueuesize" to cap the factor multiplying the MVRT sizes at the Golden Ratio
* Added some diagnostic print statements to when the queue is adjusted
* Corrected syntax errors in vetQueueSize().
* Corrected computation of new queue-size parameters.
ldmd(1):
* Version 6 upstream LDM: Added logging of a warning if sending or notifying about the
oldest product in the queue because this indicates that the product-queue might be too
small or the system too busy
* Upstream: Added diagnostic error logging to up6_run()
* up6.c: Corrected use of variable indicating status of product transmission by initializing
it
notifyme(1):
* Added "-O" option to include product origin in output
* Modified usage message to use LDM logging module
* Replaced use of synchronous LDM-5 protocols with asynchronous LDM-6 protocols in order to
greatly increase performance when used for a long time. As a consequence, this utility
will no longer work with an upstream LDM-5 server.
* Eliminated SIGSEGV if a severe upstream error causes notifyme_6() to returns NULL. It now
prints a useful error message.
pqinsert(1):
* Added ability to read a single data-product from standard input
* Added "-h" (help) option
* Enhanced man(1) page
* Added "-r" option to read the input files instead of memory-mapping them. This might be
necessary if the input files are not on local disk.
rtstats(1):
* Have connectToLdm() call ldmsend_disconnect() if clnt != NULL because the previous
assertion failed for some reason (cosmic ray?)
keep_running(1):
* Added a test for the executable's existance
showSettings(1): Corrected bug in setting of default LDMHOME value
NOAAPort:
* Configured the UDP socket for sharing the packets with other sockets and logging a
warning if it couldn't
* Added sh(1)-based noaaportBlender(1) to keep blender(1) and noaaportIngester(1) processes
running
* blender(1):
* Frame is now output when the wait-time expires even if no subsequent frames arrive
* "Starting product" INFO-level log message now includes source field from frame header
blender(1):
* Changed Key::operator<() to handle the data server being switched
* Corrected comparison of frame keys
* Renamed CircFrameBuf to FrameQueue
* Improved error messages
* Fixed SIGSEGV bug by ensuring that the key of the oldest frame in the queue is used
before the frame is destroyed
* Enabled logging of diagnostic information for a frame that arrives too late to be included
* Added a shell version of the noaaportBlender(1) Python version
Documentation:
* Feedtype webpage: Changed "FT31" to "FT30" for the "ANY" feedtype
DevOps:
* Added "workflow_dispatch" to "on" entry
* Have NOAAPort makefile print head of ingest log file to aid GitHub Actions debugging
6.14.5 2023-02-16T11:40:06-0700
ldmadmin(1):
* Made the determination of local vs. remote LDM ports in the command "ldmadmin printstats"
more robust. In particular, it now counts transitory connections due to local rtstats(1)s
when they exist.
plotMetrics(1):
* Improved some plot titles to clarify what they represent
rtstats(1):
* Commented code about what function logs messages because the code is convoluted
* Removed trailing newline in rtstats(1) log message to eliminate extraneous log message
* Eliminated sending of NULLPROC before closing connection because it's superfluous
* Documented return value of ldmsend_statsbin()
* Eliminated possibility of overrun in formatting report
* Eliminated unnecessary log message
* Replaced abort() due to unsupported remote LDM version with log_error() and ECONNABORTED
return value
* Restructured code
* Decreased the coupling between the statistics gathering module and the LDM reporting
module
* Renamed several functions to increase clarity
* Moved that which could and should be done before sending reports from inside the
report-sending code to an initialization step (e.g., MD5 checksum allocation, product
class initialization, sequence number, HIYA)
* Clarified logic by breaking complex functions into multiple functions
* Improved internal documentation
* Eliminated small memory-leak due to product class return value from HIYA not being freed
* man(1) page: Corrected description of number of bytes, number of products, and
peak-latency parameters: they are not for "this hour" but since rtstats(1) was started
Documentation:
* Removed empty entry from list of general platform requirements
* Added requirement that "RemoveIPC" in "/etc/systemd/logind.conf" must be "no" to
prevent systemd(8) from deleting the LDM's shared-memory segment that contains the
upstream LDM database.
* CHANGE_LOG:
* Consolidated 6.14.4.* entries into 6.14.4. Should have done that earlier.
6.14.4 2023-01-26T11:03:17-0700
ldmd(1):
* Eliminated log messages when an upstream LDM process that was ACCEPTed terminates. This
should reduce the number of log messages on the rtstats(1) mothership.
* Moved setting of "hiyaCalled" to top of HIYA service functions
* Made ECONNRESET error a single INFO-level log message
* Replace some ERROR log messages with FATAL ones in the main module.
NOAAPort:
* Added entries with new PDTNs to GEMPAK tables:
* g2varswmo.tbl:
000 002 000 002 Wind Direction Degree true DRCT 0 -9999.00
* g2varsncep.tbl:
001 001 194 009 Prob precip exceeding FFG % PEFG-- 0 -9999.00
rtstats(1):
* Made error reporting more robust
Documentation:
* More hyperlink corrections
* Returned Coverity Scan badge to top-level, version-specific index.html
Misc:
* Removed lint found by Coverity Scan
* ErrObj module
* Created
* Created log_dispose(ErrObj*)
* Might get used; might not.
DevOps:
* Enhanced "pre-commit" hook to vet the version ID in CHANGE_LOG and ensure a timestamp
in CHANGE_LOG when committing to the main branch
* Enhanced "OnPush.yml":
* Vet the version ID in CHANGE_LOG and "configure.ac"
* Update the "versions.html" file for LDM documentation on the artifacts server
* Deleted "--disable-root-actions" from Coverity Scan target
* Added deletion of /tmp/pqact_test.conf before creation due to /tmp having the "restricted
deletion" bit set
* Deleted use of CodeQL in OnPush.yml in favor of using Coverity Scan because it's better,
IMO
6.14.3 2022-12-22T13:34:03-0700
Documentation:
* Fixed more links in HTML documentation
6.14.2 2022-12-20T15:32:36-0700
Documentation:
* Added missing graphic "product-queue.svg"
* Fixed links in HTML documentation
6.14.1 2022-12-19T12:40:42-0700
Installation:
* Eliminated the configure(1)-script option "--disable-root-actions". Installation now
proceeds as if this option were always specified. The "make install" step now prints a
message at the end stating that the superuser (i.e., "root") must now execute the command
"make root-actions". This was necessary in order to migrate the continuous-integration
workflow to GitHub actions: autoconf(1) 2.71 on the Ubuntu 22.04 platform creates an
invalid configure(1) script if the autoconf(1) configuration-file contains this option.
Installation from the unpacked distribution now comprises the following steps (details
omitted):
- Execute the configure(1) script as the LDM user
- "make install" as the LDM user
- "make root-actions" as root
See the HTML documentation for details.
* The configure(1) script now requires that a C++ compiler be found if the "--with-noaaport"
option is specified
ldmd(1):
* Demoted the logging level of some common connection losses to reduce the number of ERROR
messages in the log file
* Improved log message when select() fails
* Improved logging and simplified up6_run()
* Only close file-descriptors 0, 1, & 2 when daemonizing
* Added "-h" (help) option
* Moved daemonization to before the creation of the shared upstream LDM database
* Ensured that the LDM server is the process group leader.
regutil(1):
* Added "-l <logdest>" option to accomodate GitHub actions
pqing(1):
* Eliminated compilation error in "tty.c" under MacOS by including <sys/ioctl.h>
NOAAPort:
* blender(1):
* Added frame-header sequence-number to determination of frame ordering.
* Corrected determination of frame ordering
* GEMPAK Tables: Added missing PDTNs
LDM registry:
* Ensured that the file-descriptor of the registry is close-on-exec
Logging:
* In order to accomodate GitHub actions, changed the determination of whether or not to use
the standard error stream from whether of not the process was a daemon to whether or not
the standard error stream is open on /dev/null
rpc(3) library:
* Eliminated compilation errors under MacOS by replacing bzero(), bset(), and bcmp() with
POSIX equivalents
* Demoted the logging level of some common connection losses to reduce the number of
ERROR messages in the log file
Documentation:
* Changed the download site to downloads.unidata.ucar.edu and eliminated the FTP site
* Added recommended home directory /home/ldm or /opt/ldm
* Changed the installation instructions to "make install" by the LDM user and then "make
root-actions" by the superuser
* Removed the Coverity Scan badge from the top-level, version-specific "index.html" page
because the code is now being scanned by GitHub's CodeQL
* Changed the example LDM home-directory from "/opt/ldm" back to "/home/ldm" because that's
more appropriate for an LDM *user*
DevOps:
* A push to the "devel" and "master" branches on GitHub now causes "make distclean" to be
executed on 2 Ubuntu systems and the latest MacOS system and the code to be scanned by
CodeQL.
* Modified Git hook "pre-commit" to ensure that the version ID in "configure.ac" is the
same as the first word in "CHANGE_LOG"
* Removed target "consistent" and its dependencies from "dev-makefile"
* Corrected code faults found by GitHub's CodeQL (mostly in GEMPAK library)
* Modified "make check" in regutil/ to accomodate GitHub actions
* Modified include of "ldm.h" in "fix_clnt.pl" to accomodate GitHub actions
* Enabled everything in OnPush.yml for LDM release
6.13.17 2022-10-24T15:01:17-0600 (should have been 6.14.0 due to the addition of blender(1))
Installation:
* Restored setting of LDM_PORT in distribution's registry
* Added support for use of chronyd(8) instead of ntpdate(8)
ldmadmin(1):
* Improved:
* Waiting for the LDM to start
* Determining if the LDM is running
* "ldmadmin start" verifies that
* The product-queue exists
* ldmd(1) owned by root and setuid
* Added support for use of chronyd(8) instead of ntpdate(8)
* Added "-o <template>" option to "plotmetrics" command to write metrics plots to files
instead of to the X-server.
* Added 1 s timeout to ldmping(1) in isRunning() to speed up "ldmadmin start" and
"ldmadmin stop"
Top-level LDM server, ldmd(1):
* Ensure printing of process ID in daemonize()
* Always make final process a group leader
* Improved daemonization of the LDM by fork(2)ing twice and redirecting all standard I/O
streams to /dev/null
* Now becomes owned by root before sending SIGTERM to process group to ensure that even
setuid processes that are owned by root receive the signal. Relinquishes root privileges
immediately after sending the signal.
scour(1):
* Documented in man(1) page:
* Missing options
* Specification of hours and minutes in configuration-file
* Corrected format of "old" specification in
* Configuration-file
* scour(1) internal comments
logging:
* Corrected logging initialization (i.e., logi_init()) when package is configured to use
the system logging daemon.
NOAAPort:
* blender(1): Created this program to blend together TCP streams of the same NOAAPort
channel from socat(1) servers at multiple, remote sites to create a more-perfect stream.
* noaaportBlender.py: Created this program to execute (and keep running)
a blender(1) and noaaportIngester(1) pair.
* noaaportIngester(1):
* Now logs reason for termination of reader
* Corrected usage message. "-S <bufsize>" should have been "-R <bufsize>"
* Documented the meaning of bit 2^4 (value 16) in the transfer type field of the
product-definition header (according to ProductMaker, it means the product is
compressed)
* Modified "gap" reporting to support receiving data frames only: if a frame-level gap
occurs but the PDH product sequence number and data block number are contiguous, then
the missing frame(s) will be ignored and not counted.
* Moved the test for no product-specific header and a transfer type of 0 to before the
test for frame gaps because the PDH for such a frame is invalid and the gap check now
requires a valid PDH
* Continue after inability to set desired scheduling of input thread
* Log previous and current product sequence numbers and data-block numbers when a gap is
detected
* Renamed some "last..." variables to "prev..." for clarity
* Change logging level of product sequence number diagnostic when a gap is found from
NOTE to WARN to ensure visibility when grep(1)ping.
* Ensured that all messages are logged when the input closes
* GEMPAK tables: Added
000 019 235 008 Joint Fire Weather Probability % JFWPRB 0 -9999.00
000 019 237 008 Dry Thunderstorm Probability % DRYTPROB 0 -9999.00
Logging (log(3)):
* Improved determination of whether or not the current process is a daemon by testing for a
controlling terminal. This correctly handles the standard error stream being open on
/dev/null.
ulogger(1):
* Have it log everything regardless of message priority
rpc(3):
* Fix latent security bug in unexecuted code
Registry (etc/registry.xml):
* /check-time/ntpdate/servers: Changed the default value of this parameter to just
"us.pool.ntp.org" because the previous value contained defunct time-servers. The new value
will automatically take effect *only* when the LDM is installed for the *first* time:
upgrades to existing installations should set this parameter manually.
Documentation:
* Enhanced section on pqact(1) decoders in the "Programs" page:
* Corrected the HTML to make visible the paragraph on shell scripts
* Added an example shell script that duplicates the FILE action and uses the ulogger(1)
utility to log error messages
Multicast:
Modified to omit sending valid multicast packet if environment variable
FMTP_INVALID_PACKET_RATIO is negative.
6.13.16 2021-09-14T09:18:55-0600
build:
* Improved handling of supplied rpcgen(1) output files on systems that
don't have rpcgen(1) installed
ldmadmin(1):
* Made vetQueueSize() print a message only when necessary
DevOps:
* Documentation:
* Removed configure(1) variables from "html/basics/index.html.in"
and renamed it "index.html"
6.13.15 2021-08-11T14:52:31-0600
configure(1):
* Made c99 the default compiler on SunOS systems because gcc(1) either
caused the configure(1) script to fail or failed to compile some files
scour(1):
* Replaced getline(3) with mygetline(3) to accommodate Solaris
* Eliminated defects found by Coverity Scan
Misc:
* mygetline.c: Added "#include <sys/types.h>" to ensure "ssize_t"
* Removed lint found by Coverity Scan
Multicast:
* fmtpRecvv3.c: Remove minor clang(1) lint
DevOps:
* dev-makefile:
* Improve rules for "coverity-scan"
* Ensure gcc(1) used
* Make more robust
* Added "reset" target
6.13.14 2021-07-27T17:24:00-0600
configure(1):
* Defined _XOPEN_SOURCE to be 600 on and only on SunOS systems in order
to support the C compiler; otherwise, it's 700
* rpcgen(1):
* Require on MacOS system
* Use if possible; otherwise, use package-supplied rpcgen(1) output
files
ldmadmin(1):
* Changed match on "ESTABLISHED" in netstat(1) output to "ESTAB" just in
case some netstat(1)'s use that instead
* Corrected error-message in "addmetrics" command
* Added missing "}" to 'while ($_ = $ARGV[0]) {'
* Documented the two types of "pqactcheck" invocations in the
manual-page
* Made more(1) the default pager for "ldmadmin log". The environment
variable PAGER will override.
* Created "showsettings" command to print information on the LDM
installation to assist troubleshooting
logging:
* Added LOG_LEVEL_FATAL and associated functions/macros
* Merged logging levels ALERT, CRIT, & EMERG into just FATAL
pqcreate(1):
* Fixed manual-page regarding the computation of the default number of
product slots
* Corrected approximate mean product-size from 51 kB to 140 kB. This
value is used when the number of slots registry parameter is
"default".
pqact(1):
* Enhanced log message about pqact(1) being at the oldest end of the
queue to include waking up to discover its position in the queue is
older than the oldest data-product. The word "oldest" still appears in
the warning -- so you can still search on it.
* Changed the logging-level associated with the "-log" option of the
FILE and STDIOFILE actions from INFO to NOTE to ensure that such
messages are always logged regardless of the logging-level of the
pqact(1) process
* Removed or demoted to DEBUG some redundant ERROR messages related to
the inability to pipe to a decoder
* Fixed bug in which a pattern/action entry wouldn't have its time of
last-use reset if it was already at the head of the most-recently-used
list. If this situation persisted for 6 hours, then the entry would
be removed from the list and its file descriptor would be closed.
This latter situation could only occur if one entry monopolized its
pqact(1) process for 6 hours.
* Modified the "-strip" option: it no longer deletes characters whose
unsigned value is greater than 127. This allows textual products in an
extended ASCII character set (e.g., ISO 8859-1).
scour(1):
* Replaced scour(1) script with multi-threaded C program that scours
much faster than the script
* Added scour configuration-file to LDM registry
* Renamed "scour" script to "scour.sh" in case users still want to use
it
* Added "-d" option to scour(1) man-page
* Increased maximum size of pattern field to MAX_INPUT bytes
* Function replacements to accommodate SunOS 5:
* getline(3) -> mygetline(3)
* fstatat(2) -> lstat(2)
rpc(3) library:
* Replaced call to bindresvport() in clnttcp_create() with binding to an
ephemeral port to eliminate the necessity for having root privileges
LDM Registry:
* Added SCOUR_EXCLUDE_PATH for pathname of file that lists directories
to be excluded by scour(1)
Misc:
* Removed much lint identified by clang(1)
* Removed lint identified by Coverity Scan
Documentation:
* Removed reference to installing from an RPM because it doesn't exist
* Updated the instructions on using metrics to monitor the LDM
* Remove obsolete feedtype webpages
NOAAPort:
* GRIB2 (GEMPAK) Tables:
* README: Modified the "clone" command to work-around permission
problems
* Added:
* Ice Thickness (ICETK, (10,2,1,0))
* Rip Current Occurrence Probability (RIPCOP) (10,1,4,0)
* Total Water Level Accounting for Tide, Wind and Waves (TWLWAV)
(10,3,205,0)
* Mean Increase in Water Level due to Waves (SETUP) (10,3,207,0)
* Time-varying Increase in Water Level due to Waves (SWASH)
(10,3,208,0)
* Erosion Occurrence Probability (EROSNP) (10,3,252,0)
* dvbs_multicast(1): Replaced non-standard usleep(3) with standard
nanosleep(3)
* noaaportIngester(1):
* Created "-R <buf_size>" option to set the size of the receive
buffer in order to reduce the number of missed packets.
(The operating system must allow the new size.)
* Ensure that the NEXRAD feedtype is assigned to every product whose
parameter "product-specific type" in the product-specific header
identifies it as being NEXRAD
Multicast:
* fmtpRecvv3(3):
* Corrected logic bug in tracking last multicast product index
* Ignore FMTP PDU if header flag is unknown
* Log closing of retransmission socket and ensure set to -1 on error
* Made handling of retransmitted duplicate BOP more robust:
* retxBOPHandler() and BOPHandler(): Made return-value indicate
if BOP was invalid or a duplicate
* retxHandler():
* Modified to ignore an invalid BOP
* No longer relies on a positive product-size to indicate a
tracker-map entry
* Exception throwing on no tracker-map entry replaced with
logging a notice
* Created McastProdPar which encapsulates multicast
product-index and sequence number object under mutex
protection
* Improved log message, documentation, and internal commentary
* Improved handling of last product whose BOP needn't be
requested
* fmtpRecvv3.cpp: Replaced <inttypes.h> with <cinttypes> to get
PRIu32 macro
* Improved debug messages
* UdpRecv.cpp:
* Corrected reversed "expected" and "actual" values in error message
about a too-small payload message
* IperfFmtpPdu(1): Created to create invalid FMTP packet for malicious
testing by iperf(1)
* mldm_sender(1): Removed lint
* recv_test(1): Corrected documentation & simplified diagnostic
statement
* configure.ac & FmtpConfig.h.in: Added _XOPEN_SOURCE
* FmtpConfig.h{,.in}: Removed from repository
* UdpSend(3): Created private BlackHat class for sending invalid packets
based on the environment variable FMTP_INVALID_PACKET_RATIO
* Created three levels of message authentication of multicast packets
dependent on the value of environment variable FMTP_MAC_LEVEL:
- unset or "0": No MAC
- "1": SHA256 HMAC
- "2": Ed25519
* Up7Down7_test: Increased sleep after product insertions from 1 s to 2
s to give Ed25519 more time
* Improved FMTP packet structures regarding I/O handling and padding:
* Deleted RETX_REQ_LEN
* Replaced flexible arrays with [1] to accommodate g++ 8.3
* Improved BOPMsg structure
* Eliminated use of sizeof(struct ...) where appropriate
* Converted FMTP PDU size from a compile-time to a runtime parameter
DevOps:
* Added "regpar.tab" to configured files
* Changed "388" to "@LDM_PORT@" in registry
* publish(1):
* Eliminated deletion of tarballs with the same major and minor
numbers but lower bug-fix numbers because some sites -- rather
than implementing their own repository -- depend on older versions
always being accessible upstream.
* Improved messages
* .travis.yml:
* Enabled "branch_pattern: devel"
* push(1):
* Tag the commit if and only if it's on the main branch
* Execute autoreconf(1) after setting a new version identifier in
"configure.ac"
* Doxygen:
* Doxyfile.developer: Set output directory to "doxygen-output"
* Delete obsolete parameters
* .git/hooks: Created pre-commit(1) & pre-push(1)
* UXF diagrams: Improved and added to repository
* dev-makefile(5): Created
* check(1): Created
* scan(1): Created
* configure.ac:
* Improved setting of CFLAGS, CXXFLAGS, AM_CFLAGS, & AM_CXXFLAGS
* Removed "pqcreate.1" and "regpar.tab" from AC_CONFIG_FILES
* deployAll(1):
* Added version argument
* Modified to use an associative array and to print asynchronous
logs at the end
* scour/:
* testCscour.py(1): Tests scour(1) correctness
* aSpringTree(1): Builds a tree of depth and breadth for testing
scour(1)
* benchMarkIt.sh(1): Tests performance of scour(1)
* .gitignore: Deleted inappropriate slash prefix of "scour.conf"
* pq/Makefile.am: Moved HAVE_CUNIT to before test_PROGRAMS to enable
"make check" on systems with no CUnit package installed
* Makefile.am: Added CLEANFILES, DISTCLEANFILES, and
MAINTAINERCLEANFILES
* pqact/Makefile.am: Decreased time for "make check"
6.13.13 2020-11-03T15:22:01-0700
ldmadmin(1):
Switched the order of the bin/ and util/ directories when setting PATH
because one site had a script named "util/ldmd".
ldmd(1):
* Increased the size of the listen(2) queue from 32 to 1024 in order to
support the Unidata LDM that's accepting rtstats(1) messages
NOAAPort:
* GEMPAK Tables: Added new WMO parameters:
000 000 000 010 Temperature
000 002 000 006 Wind Direction
000 002 022 002 Wind Speed (gust)
000 006 001 002 Total Cloud Cover
000 006 013 005 Ceiling
010 000 003 002 Sig Hgt Wind Waves and Swell
* GEMPAK NSSL parameter table:
* Added MRMS parameters that just appeared in NOAAPort. See
<https://www.nssl.noaa.gov/projects/mrms/operational/tables.php>:
209 002 005 000 Lightning Prob. next 30min Grid % LgtProb30min 0 0.00
209 006 030 000 MultiSensor QPE 01H Pass1 mm MrgdQpe1HP1 0 -1.00
209 006 031 000 MultiSensor QPE 03H Pass1 mm MrgdQpe3HP1 0 -1.00
209 006 032 000 MultiSensor QPE 06H Pass1 mm MrgdQpe6HP1 0 -1.00
209 006 033 000 MultiSensor QPE 12H Pass1 mm MrgdQpe12HP1 0 -1.00
209 006 034 000 MultiSensor QPE 24H Pass1 mm MrgdQpe24HP1 0 -1.00
209 006 035 000 MultiSensor QPE 48H Pass1 mm MrgdQpe48HP1 0 -1.00
209 006 036 000 MultiSensor QPE 72H Pass1 mm MrgdQpe72HP1 0 -1.00
209 006 037 000 MultiSensor QPE 01H Pass2 mm MrgdQpe1HP2 0 -1.00
209 006 038 000 MultiSensor QPE 03H Pass2 mm MrgdQpe3HP2 0 -1.00
209 006 039 000 MultiSensor QPE 06H Pass2 mm MrgdQpe6HP2 0 -1.00
209 006 040 000 MultiSensor QPE 12H Pass2 mm MrgdQpe12HP2 0 -1.00
209 006 041 000 MultiSensor QPE 24H Pass2 mm MrgdQpe24HP2 0 -1.00
209 006 042 000 MultiSensor QPE 48H Pass2 mm MrgdQpe48HP2 0 -1.00
209 006 043 000 MultiSensor QPE 72H Pass2 mm MrgdQpe72HP2 0 -1.00
* Removed
209 002 004 000 Lightning Probability 0-30min % LgtProb30min 0 -9999.00
because it's not in NSSL's table and because it's long-name is too
similar to and it's short-name is identical to (209,002,005) above
from NSSL.
pqact(1):
* Hopefully closed a tiny window-of-vulnerability in which a SIGALRM
might be generated after a successful write(2) to a (very) slow
decoder but before the alarm is dismissed -- leading to the abnormal
termination of the process. This has occurred at more than one site.
* Set default SIGALRM handing to "ignore" just in case
* Documented "-t <timeout>" option for piping to a decoder
* Added decoder command to pbuf_flush() log messages
Multicast:
* Up7.c:
* Made auto-initialize: changed public up7_init() to static init()
* ChldCmd.c:
* Demoted log messages from ERROR to NOTE because the message
"Success" from the child process isn't an error
* Improve handling of OpenSSL:
* Move library examination from LDM's to FMTP's configure.ac
* Create FmtpConfig.h.in
* Add declaration of EVP_PKEY_new_raw_private_key() if necessary
* fmtpRecvv3:
* Made handling of index of last multicast product more rigorous
Developer:
* publish.sh(1):
* Converted to functional form. Needs to be tested.
* Removed FTP references when modifying "versions.inc" on website
* Top Makefile:
* Added targets "push-gempak-tables" & "pull-fmtp"
* push(1):
* Made agnostic of branch
* Replaced use of "git commit <file> ..." with use of "git add"
* .travis.yml:
* Made Coverity Scan act on all branches
* deployAll(1):
* Replaced use of internal shell function with calling deploy(1)
* Made all LDM7 deployments asynchronous
* configure.ac:
* Added "-I" and "-L" checks for /opt/openssl and
/usr/lib64/openssl11
6.13.12 2020-09-30T10:25:12-0600
Installation:
* Added EOF newline to regpar.tab.in to accommodate Solaris' sed(1)
ldmadmin(1):
* plotmetrics:
* Corrected sense of local/remote port connections. They were
reversed all this time.
* Put $LDMHOME/util at head of PATH to support customization
* Changed URL for GEMPAK tables updates from "git@github..." to
"https://..." because the former could require a key
ldmd(1):
* Top-Level Server:
* Eliminated superfluous error messages at process termination about
the upstream LDM database not existing if the "-n" (do nothing)
option is specified.
* Replaced log_log_q() with loc_add() & log_flush() in "Child ...
exited with status ..." messages.
* Combined hostname and dotted-quad "Denying connection from ..."
messages into one
* Upstream LDM:
* Don't terminate previously-executing upstream LDM with a different
protocol version number.
* one_svc_run(): Replaced use of select() with poll()
* Increase service timeout from 60 s to 120 s
* Downstream LDM-6:
* Replaced use of IS_ALIVE with fixed timeout of 5 minutes
* Created vetProduct() and modified down6_hereis() and
down6_comingsoon() to use it.
* Changed logging level of "too-old" and "unrequested" products from
INFO to WARN in order to unconditionally reveal why products
aren't being inserted into the product-queue due to a slow
connection (because it's being throttled, for example). If this
happens to you, then you'll need to work with your network
administrator to allow LDM traffic to pass unhindered. NB: This
means a cold-started LDM might log some "too-old" warning
messages, initially.
* Log a warning that the (PRIMARY/ALTERNATE) transfer-mode won't
work correctly if there are more than two REQUESTs for the same
feed.
* Modified the algorithm for determining when to switch between
PRIMARY and ALTERNATE transfer-modes to consider whether or not
the same feed is being received via multicast. This will only have
an effect if multicast reception is enabled -- which isn't by
default.
pqact(1):
* pqact.c:
* Simplified decision on whether or not to write a new state-file by
using tvIsNone() on the insertion-time of the last, successfully-
processed product
* Ensure valid time-of-last-insertion before writing state file at
exit
* Correct parsing/formatting of state file with negative seconds.
* Modified "Starting Up" message to log the command-line
* Changed log message at process termination from
Behind by %g s
to either
No product was processed
or
Last product processed was inserted %g s ago
to better reflect the meaning
* Replaced log_error_q() with log_add() & log_flush().
* filel.c:
* Added 6 hour timeout for inactive entries that, nevertheless, had
open file descriptors to eliminate the possibility of pqact(1)
holding open thousands of files that have been scoured and,
consequently, filling up the disk with invisible files. This also
improves the time it takes for the LDM to shut down.
* Corrected truncation of output file by the FILE action when both
"-overwrite" and "-metadata" are specified. Before, the file would
be truncated to the size of the data-product even though it now
has a metadata header. Thanks to Bryan for spotting and analyzing
this.
* Made the removal of the least-recently-used action (and the
comcomittant closure of its open file descriptor) repeat until the
current action can succeed or a failure occurs
* Added use of the above if a child process can't be fork(2)ed for a
decoder
* Improved log messages for when an action with an open file
descriptor is deleted from the list of such actions and its
descriptor is closed
uldb.c:
* Removed lint found by Coverity Scan
NOAAPort:
* Changed pre-installation instructions to recommend disabling UDP
packet sequence checking.
* Created keep_running(1) script to ensure that NOAAPort ingesters are
restarted if they crash. Added examples to NOAAPort documentation.
* Updated main page of Doxygen-based documentation. In particular,
corrected discussion of UDP packet reassembly and setting of system
parameter net.ipv4.ipfrag_max_dist to zero.
* Added new parameters to the NCEP GEMPAK tables:
* 000 001 228 010 Flat Ice Accumulation (FRAM) kg m**-2
FICEAC 0 -9999.00
* 010 003 204 000 Freezing Spray -
FRZSPR 0 -9999.00
* 000 002 231 000 Tropical Wind Direction Deg
TPWDIR 0 -9999.00
* 000 002 232 000 Tropical Wind Speed m s-1
TPWSPD 0 -9999.00
* 000 001 234 008 Precipitation Duration hr
PCPDUR 0 -9999.00
* 000 002 009 000 Vertical Velocity (Geometric) m s**-1
DZDT 0 -9999.00
* updateGempakTables(1):
* Changed the "make install..." to "mv -f *.tbl..." to make it
independent of the build mechanism.
* NOAAPortIngester(1):
* Changed logging-level of messages for first fragment of GOES
imagery from NOTE to INFO.
* Modified thread-creation to prevent the program from immediately
terminating if the RHEL Insights Client security package is
running. That package will, by default, prevent the creation of a
thread with non-default scheduling -- even if the process is owned
by root. The program will now try to create threads with
high-priority, real-time scheduling but will create default
threads and log a warning if it can't. The default threads will
increase the risk of missing NOAAPort packets.
* process_prod.c:
* Added feed to notice-level log message about inserted product
* Makefile.am:
* Replaced ingest-verification files with ones containing new log
messages
Logging:
* Library:
* Created macro log_log(level, ...)
* Replaced minimum field-width in default log messages with
canonical column-start positions to reduce line lengths and
CPU usage
* Improved concurrency:
* Unlocked file and mutex if thread cancelled. Irrelevant
for LDM6.
* Added automatic deletion of message queue when thread
exits, so calling log_free() is no longer necessary.
* Added use of pthread_atfork(). Irrelevant for LDM6.
* Added "Terminating logging" message at DEBUG priority
* refresh_logging(1):
* Replaced "command kill" with "/bin/kill" to ensure correct
behavior
* Documented internally how log_refresh() will cause the log file to be
closed/opened *on the next log message* and not before
Product-Queue (pq.c):
* Changed return values of pq_sequence() and pq_sequenceLock()
* Documented that pq_sequence() and pq_sequenceLock() error codes are
negative
* Improved error-message in pqe_newDirect()
* Added error-message to pqe_discard()
* Corrected return status of ctl_rel() on mtof() failure
Documentation:
* Troubleshooting webpage:
* Removed the incorrect value of 4096 for the average product size
when computing the number of product-slots to create in a new
product-queue.
Misc:
* Removed lint found by Coverity Scan
* ldm_formatCmd()
* Created to format a command-line from a command vector
* ghostname():
* Improved logic in for determining the fully-qualified name of the
local host, which is used by rtstats(1). The LDM registry is still
checked first, then gethostname(), and then the default value set
by the configure script. If, after all this, the name still isn't
fully qualified, then a system call will be made in an attempt to
make it fully qualified. This means that the "hostname" parameter
in the LDM registry needn't be set unless ghostname() doesn't
return a fully-qualified hostname or a different name is desired.
* ldmfork(3):
* Documented `errno` values
* Improved internal commentary
Multicast:
* ChildCmd.c:
* Ensured that all pipe(2) file descriptors in the child process are
greater than 2 to avoid conflicts when duplicating them to the
standard file descriptors (i.e., 0, 1, and 2).
* Product-Queue (pq.c):
* Disable thread-cancellation on entry and revert on return
* pq_sequenceHelper():
* Print number of locked products
* The product will now be unlocked upon return if
* The offset argument is NULL
* The product doesn't match the class specification
* An error occurs, either in the product-queue or the
caller's function
* Added missing decrease of lock count
* OESS scripts:
* Modified to handle different virtual circuits that, nevertheles,
have the same switch name, port name, or VLAN tag
* Corrected URL for OESS services
* up7.c:
* Simplified and improved robustness by eliminating second
server-side transport.
* Cleaned-up subscription-request logic. Ensured that an error
message is logged if anything goes wrong.
* request_product_7_svc():
* Added call to svcerr_systemerr() if there's no current
subscription
* Added test_connection_7_svc() from Down7
* Improved handling of client destruction when subscribing LDM7
disconnects
* Added product-index to "Sending missed product" log message
* Squashed SIGSEGV when getting number of product-queue slots for
product-index map by always opening product-queue
* Corrected bug in determining PIM-file parent directory
* Replaced use of global product-queue with one given to up7_init()
* Made up7_init() and up7_destroy() public
* UpMcastMgr.c:
* Added check of logging destination for mldm_sender(1)
execution
* Assert that the logging module doesn't hold a mutex lock
before calling fork()
* Stop multicast LDM sender, mldm_sender(1), during multipoint
VLAN reconfiguration
* down7.c:
* Increased sleep duration between subscription attempts from 30
seconds to 120 seconds
* Increased RPC subscription timeout to 10 minutes
* Specified absolute pathname of vlanUtil(1) to workaround Ubuntu's
non-standard execvp(2)
* Replaced reliance on getting the RPC timeout from the client
handle to explicitly setting the subscription-timeout because the
RPC timeout is unset until the first call to `clnt_call()`
* Included RECEIVE entry in computation of count of number of
upstream suppliers of a given feed
* Added mutex loccking/unlocking in down7_init() to silence Coverity
Scan
* Moved test_connection_7_svc to Up7
* Added periodic heartbeat to upstream LDM7
* Improved some log messages
* MldmRcvr.c:
* eop_func(): Improved INFO message
* missed_prod_func(): Added flushing of errors
* ProdNotifier.cpp:
* startProd(): Improved error-message
* missedProd(): Added error-message
* MldmRcvrMemory.c:
* Removed lint found by Coverity Scan
* Replaced searching of YAML mapping with streaming
* Improved commentary
* FMTP:
* Got multicast message authentication working
* Modified logMsg(std::string) to always log at NOTE level
* fmtpSendv3:
* Improved debug logging
* Added release of products held by a unicast socket that just
disconnected to prevent all products in the queue from
eventually being locked
* Release products held by socket that just disconnected
* Eliminated deadlock
* Added rcvrCount()
* Added log_info() for duplicate BOP
* Initialized numRetrans to -1
* TcpSend:
* Explicitly enabled SO_REUSEADDR and SO_KEEPALIVE
* fmtpRecvv3:
* Moved direct sending of product-EOP to sender on unicast
reception thread to adding message to retransmission queue on
that thread to obviate deadlock between that thread and
sender's retransmission thread.
* Improved handling of missed multicast packets
* Adapted to use UdpRecv
* Modified to only consider valid FMTP messages
* OffsetMap:
* Added missing deletion of entry by om_get()
* Modified OffsetMap::get() to indicate no-entry instead of
throwing an exception
* UdpRecv.c:
* Created
* mldm_sender(1):
* Improved releasing of product in product-queue:
* Added product release on multicast failure
* Added debug message about releasing locked product
* Added registry parameter "/fmtp-retx-timeout" for FMTP timout in
seconds
* Modified to use registry parameter "/fmtp-retx-timeout"
* Modified "-r <fmtp_timeout>" option to be in seconds
* Merged mls_tryMulticast() into mls_startMulticasting()
* Added error message on fmtpSender_send() failure
* Don't multicast product if no FMTP receivers
* Modified mcastProd() to return LDM7_NORCVR and not explicitly
release product
* Eliminated double free()ing of `mcastInfo`
* Continue if no receivers
* Log more errors
* Demoted "Product not multicast because no receivers from INFO to
DEBUG
* Added notice-level log message (with status) about terminating
normally
* Up7Down7_test(1):
* Added (commented out) single, small product possibility for
debugging
* vlanUtil(1):
* Replaced use of gawk(1)-only bit operations with shell's for
portability
* Corrected setting of broadcast address string
* setMcastRate(1):
* Created to set the multicast sending rate. Uses tc(1). Must be
executed by root.
Developer:
* Removed publishing rules from makefiles in favor of using "publish.sh"
* send_recv_test.h:
* Improved commentary on multicast addresses
* configure.ac(1):
* Added LDM_BIN_DIR
* push(1):
* Replaced "commit -a" with list of modified files
* Replaced "git push" arguments with "--mirror"
* Instrumented OffsetMap
* Only deploy to NWSC node6
* ldm7PreInstall:
* Added perl and corrected g++ package-name (gcc-g++)
* OffsetMap.cpp:
* Demoted get() log message from NOTE to DEBUG
* pq.c:
* Demoted pq_sequenceHelper() log message from NOTE to DEBUG
* Added "delayQueue/" to Git.
* Corrected target "push-%" in top-level makefile
* Removed Authorizer_test from "make check" because GTest segfaults at
exit.
* pthread_atfork_test.c:
* Created
* one_svc_run():
* Converted timeout argument from unsigned to int to support
indefinite timeout
* Down7::up7Proxy_heartbeat():
* Fixed setting of `lastRequest`
* send_test(1) & recv_test(1):
* Added "-g <grpAddr>" option
* Made independent of LDM library
* Changed HELLO_GROUP to 234.128.117.1 (unicast prefix based)
* Added to distribution (but neither built nor installed)
* deployAll(1): Enabled California and Maryland
* Disabled "make installcheck" in "mcast_lib/ldm7" on Travis CI
* pq_test_insert(1):
* Increased inter-product gap to 0.1 s
* Moved sleep from beginning to end of insertion-loop
* deployAll(1):
* Added ldm7@idc-uva.dynes.virginia.edu
* Changed deployment to uni15 to deployment to uni17
* Top Makefile.am:
* Removed superfluous "muticast-check" target
* .travis.yml:
* Added cat(1) of top-level Makefile on error