Logger and extractor of time-series data (e.g. EPICS PVs).
- Data are objects, indexed in time order.
- Supported Control Infrastructures: EPICS, ADO, LITE. Easy extendable.
- Wide range of data objects: strings, lists, maps, numpy arrays, custom.
- Data objects could be inhomogeneous and have arbitrary frequency.
- Self-describing data format, no schema required.
- Data objects are binary-serialized using MessagePack.
- Fast online compression.
- Fast random-access retrieval of data objects for selected time interval.
- Simultaneous writing and reading.
- Typical speed of compressed serialization to a logbook file is 80 MB/s.
- De-serialization speed is up to 1200 MB/s when the logbook is cached in memory.
- Basic plotting of the logged data.
Dependencies: msgpack, caproto, lz4framed. These packages will be installed using pip:
pip3 install apstrim
The example program for deserialization and plotting apstrim.view, requires additional package: pyqtgraph.
# Serialization of one float64 parameter from an EPICS simulated scope IOC:
:python -m apstrim -nEPICS -T59 testAPD:scope1:MeanValue_RBV
Logging finished for 1 sections, 1 parLists, 7.263 KB.
...
# The same with compression:
:python -m apstrim -nEPICS -T59 testAPD:scope1:MeanValue_RBV --compress
Logging finished for 1 sections, 1 parLists, 6.101 KB. Compression ratio:1.19
...
# Serialization 1000-element array and one scalar of floats:
:python -m apstrim -nEPICS -T59 testAPD:scope1:MeanValue_RBV,Waveform_RBV --compress
Logging finished for 1 sections, 2 parLists, 2405.354 KB. Compression ratio:1.0
...
# Note, Compression is poor for floating point arrays with high entropy.
# Serialization of an incrementing integer parameter:
:python -m apstrim -nLITE --compress liteHost:dev1:cycle
Logging finished for 1 sections, 1 parLists, 56.526 KB. Compression ratio:1.25
...
# In this case the normalized compressed volume is 9.3 bytes per entry.
# Each entry consist of an int64 timestamp and an int64 value, which would
# occupy 16 bytes per entry using standard writing.
Example of deserialization and plotting of all parameters from several logbooks.
python -m apstrim.view -i all -p *.aps
Python code snippet to extract items 1,2 and 3 from a logbook for 20 seconds interval starting on 2021-08-12 at 23:31:31.
from apstrim.scan import APScan
apscan = APScan('aLogbook.aps')
headers = apscan.get_headers()
print(f'{headers["Index"]}')
extracted = apscan.extract_objects(span=20, items=[1,2,3], startTime='210812_233131')
print(f'{extracted[3]}')# print the extracted data for item[3]
# returned:
{'par': 'liteBridge.peakSimulator:rps', # object (PV) name of the item[3]
'times': [1628825500.8938403, 1628825510.898658], # list of the item[3] timestamps
'values': [95.675125, 95.55396]} # list of the item[3] values