v0.6.0

Adds decoders for PostgreSQL btree, control and heap files. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka

Adds new option skip gaps and output binary as hex string.

Make bits/bytes formats work a bit more intuitive.

Bug fixes to to_hex, to_base64 and trim functions.

Bug fixes and additions to bson, bitcoin_transaction, mp4, id3v2, html, and matroska formats.

Changes

• bits,bytes now are real binaries and not raw decode value. This means they behave more like you would expect. #666

  # build your own strings(1)-like tool:
  # scan matches range in a binary using a regexp and output ranges as new binaries
  # \w\s looks for whitespace and alpha/numeric characters
  # uses `...` raw string literal to not have to escape
  # select/test to only include strings containing "thread"
  # dd to display with no truncation
  $ fq -d bytes 'scan(`[\w\s]{16,}`) | select(test("thread")) | dd' file.mp4
       │00 01 02 03 04 05 06 07│01234567│
  0x250│36 20 6c 6f 6f 6b 61 68│6 lookah│.: raw bits 0x250-0x262.7 (19)
  0x258│65 61 64 5f 74 68 72 65│ead_thre│
  0x260│61 64 73               │ads     │
       │00 01 02 03 04 05 06 07│01234567│
  0x260│            31 20 73 6c│    1 sl│.: raw bits 0x264-0x273.7 (16)
  0x268│69 63 65 64 5f 74 68 72│iced_thr│
  0x270│65 61 64 73            │eads    │

• to_hex,to_base64 now correctly handles raw decode values, before the raw bits would be turned into codepoints and then binary UTF-8 possibly introducing invalid codepoints (0xfffd). Thanks @Rogach #672

  $ fq -r '.uncompressed | to_hex' file.gz
  f6f2074cf77d449d
  
  # with the change to add hex bits format you can also do this now
  $ fq -Vr -o bits_format=hex .uncompressed file.gz
  f6f2074cf77d449d

• tovalue Now output a "deep" jq value, before it was shallowly a jq value which could be confusing, ex tovalue | .header could be a decode value.
• New option skip_gaps for -V/tovalue can be used to filter out gap fields when represented as JSON. Gaps are bit ranges that no decoder added any field for. #649
  • Can for example be used to remove trailing data that will end up as gap0 etc.
  • Note that gaps in an array that gets filtered out might affect index numbers.
• Add bits_format=hex to represent raw bits as hex string in JSON. #673

  # output decode tree, JSON with binaries as strings and JSON with binaries as hex strings
  $ fq '.packets[0].packet | ., tovalue, tovalue({bits_format:"hex"})' file.pcap
      │00 01 02 03 04 05 06 07│01234567│.packets[0].packet{}: (ether8023_frame)
  0x28│8c 85 90 74 b8 3b      │...t.;  │  destination: "8c:85:90:74:b8:3b" (0x8c859074b83b)
  0x28│                  e8 de│      ..│  source: "e8:de:27:c8:9a:6e" (0xe8de27c89a6e)
  0x30│27 c8 9a 6e            │'..n    │
  0x30│            08 06      │    ..  │  ether_type: "arp" (0x806) (Address Resolution Protocol)
  0x30│                  00 01│      ..│  payload: raw bits
  0x38│08 00 06 04 00 01 e8 de│........│
  0x40│27 c8 9a 6e c0 a8 01 01│'..n....│
  0x48│00 00 00 00 00 00 c0 a8│........│
  0x50│01 e6                  │..      │
  {
    "destination": "8c:85:90:74:b8:3b",
    "ether_type": "arp",
    "payload": "\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\ufffd\ufffd'Țn\ufffd\ufffd\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\ufffd\ufffd\u0001\ufffd",
    "source": "e8:de:27:c8:9a:6e"
  }
  {
    "destination": "8c:85:90:74:b8:3b",
    "ether_type": "arp",
    "payload": "0001080006040001e8de27c89a6ec0a80101000000000000c0a801e6",
    "source": "e8:de:27:c8:9a:6e"
  }

• Explicit use of d/display on a binary will now always hexdump, this feels more intuitive. Before it could output raw binary as display is used as an implicit output function. Implicit (you don't mention d at all) can still output raw binary. #665

  # outputs raw binary (if stdout is not a tty)
  $ fq -n '[1,2,3] | tobytes' | xxd
  00000000: 0102 03
  
  # outputs hexdump (even if stdout is not a tty) as we explicitly use d
  $ fq -n '[1,2,3] | tobytes | d' | cat
     |00 01 02 03 04 05 06 07|01234567|
  0x0|01 02 03|              |...|    |.: raw bits 0x0-0x2.7 (3)

• trim can now handle multi-line strings. #668
• Help texts originated from markdown are now rendered a bit more nicely and compact. #661
• Fix weirdly rendered monospace font in demo SVG when using Japanese locale and Chrome on macOS. Thanks @acevif helping with investigation. #655 #662
• Fix a bunch of typos. Thanks @kianmeng and @castilma. #644 #670

Decoder changes

• bits,bytes Is a proper binary not a raw decode value. #666
• bitcoin_transaction Properly decode witness items array. #671 Thanks @Rogach
• bson Add javascript, decimal128, minkey and maxkey support. Fix decoding of datetime and use the correct size type for binary and document size. Thanks Matt Dale @matthewdale. #650
• id3v2
  • Add WXXX frame support. #656
  • Add CTOC flags support. #657
• html Is now probeable. #667
• matroska Fallback raw is probe fail or file_data #645
• mp4
  • Better description for terminator atom (is a QuickTime thing) #651
  • ctts,infe,iinf,trun more proper decoding based on version. #643
  • Use correct QuickTime epoch for timestamps. #674
• pg_btree,pg_control,pg_heap Add PostgreSQL btree, control and heap support. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka. #415

Changelog

• d4f8dfa Add flavour arg to postgres parser
• 3c6ea87 Add heap infomask flags parser to PostgreSQL
• 0107d12 Add pgproee14 flavour to postgres
• 9a96da8 Add postgres pg_control parser
• 3b81d99 DBState enum for postgres
• 51878dc PostgreSQL heap page parser implememtation.
• 6ed0263 PostgreSQL: accept only normal item pointers
• ffc08cf PostgreSQL: add heap for pgpro14
• b4ae1d5 PostgreSQL: add pg_control for pgproee 12
• a653710 PostgreSQL: add pg_control to pgpro14
• d9de2d4 PostgreSQL: fix
• 96a86e2 PostgreSQL: fixes
• 96a96a5 PostgreSQL: heap impl for pgproee 12
• 6618e76 PostgreSQL: heap impl for version 11
• ce9ae76 PostgreSQL: heap tuples implementation
• ffd7c9b PostgreSQL: implement pgproee 14
• 850dc60 PostgreSQL: lp_flags format
• f5278f3 PostgreSQL: pg_control for ver 12
• 07056c2 PostgreSQL: pg_control impl for pgproee 10
• 972c5a3 PostgreSQL: pg_control, pgheap impl for pgproee13
• d148727 PostgreSQL: pgheap impl for pgproee10
• 2d3884a PostgreSQL: pgproee11 heap impl
• d8b891c PostgreSQL: pgproee11 pg_control impl
• b722b21 PostgreSQL: ref
• fdb3b3e PostgreSQl: fix offset
• 621c4c4 PostgreSQl: heap impl for version 13
• c273a6c PostgreSQl: pg_control impl for version 13
• 01b380e PostgrreSQl heap decode refactoring
• bebdfa9 Try to implement pgwal - fail.
• c9d3b8f Update docker-golang to 1.20.4 from 1.20.3
• 2a5927b Update github-go-version to 1.20.4 from 1.20.3
• c405afd Update gomod-golang-x-crypto to 0.9.0 from 0.8.0
• ed0cd6d Update gomod-golang-x-net to 0.10.0 from 0.9.0
• deaf5ef WalLevel for postgres
• f069ddc [WIP] initial attempt to add postgres
• 2b1bdfb add icu version mapper
• cf1e7b2 add pgpro11 for postgres
• 9570d4d add pgpro12 postgres
• 7bf6b11 add pgpro13 heap
• 2b3035f add pgpro13 to postgres
• f56c72d add postgres tests for mem, cpu profiling
• 2ee01f7 allow to change FillGaps in decoder
• a3361e7 bitcoin: fix witness item structs
• 222cd88 bits,bytes: Behave as binary instead of raw decode value
• 9a982d0 bson: add BSON test file generator module and correct BSON format docs
• 40630d3 bson: fix doc formatting and add author info
• 2017ff8 bson: support all non-deprecated types and fix int/uint bugs
• b08ef00 decode,interp: Refactor format groups into a proper struct
• af68511 dev,doc Clarify some dev docs and rename launch.json to be a template
• 97c952b doc: Add some more examples
• 88be3a7 doc: Hopefully fix svg fixed font issue
• dd4fa26 doc: fix typos
• b0e4da2 fix non-ascii characters handling in to_hex and to_base64 functions
• a4a332b formats: Clenaup naming a bit
• b3b6cd0 gzip.go: fix typo in variablename: delfate
• 2c505fe help,markdown: Rewrote and made text rendering nicer
• e2eb667 html: Add to probe group
• d010dce id3v2: Add WXXX (desc/url) frame support
• f237db2 id3v2: Decode CTOC flags
• 684a083 interp,decode: Support decode group argument
• 8a468f4 interp: Add hex bits format
• c512713 interp: Add skip_gaps option for tovalue/-V
• 033498b interp: Don't output raw binary if display is called explicitly
• ee66fec interp: Make tovalue output behave as jq value
• d5ae116 interp: trim: Add multi-line support
• 8d317dd lsn mapper
• fcd7fbc mappers for postgres
• 8941b13 matroska: file_data: Fallback to raw if probe fails
• 7adc1e7 mp4: Better description for QuickTime terminator atom
• 493848a mp4: Use correct epoch for quicktime timestamps
• 3c6d31b mp4: ctts,infe,iinf,trun: More ISOMFF version handling
• d6f785c pcap: Add forgotten help test
• 8cf83fb pg_control implementation
• 441ecee pgpro version mapper
• 5ff6273 postgres: add additional checks in pg_heap
• 4676590 postgres: add argument to calc page's check sum correctly
• 7fd4109 postgres: add btree index tests
• 1aa08e3 postgres: add btree, pg_control to how_to.md
• 2423f86 postgres: add how_to.md - how to generate test files for postgres
• 97bbc22 postgres: add page arg in pg_btree, change args names in pg_heap
• 6aed238 postgres: add pg_wal for pgproee11 as copy of postgres14
• 08eb303 postgres: add postgres format docs, refactoing postgres flavours
• 90386a6 postgres: add postgres.md to format
• 09c42c3 postgres: add state to wal struct
• c2591ac postgres: add test data with specific values
• de68785 postgres: add test files
• 4db1284 postgres: add tests
• 3c7ed5d postgres: add tests data
• e66baa7 postgres: add wal checks
• fb7778a postgres: add wal tests
• 9f61e63 postgres: allow all flovours to decode btree index
• ccf2edb postgres: better versions probing in pg_control, fix holes, better tests
• f3f259a postgres: btree add free space
• efda7b3 postgres: btree handle full file
• 87b7acf postgres: btree impl
• d370f5d postgres: btree impl
• e839191 postgres: btree refactored by Mattias Wadman
• 9f1adb2 postgres: change AssertPosBytes to AssertPos (bits)
• 3e09f9f postgres: change tuple struct in heap
• e6a9cdb postgres: doc
• 6281b50 postgres: exclude wal tests for now
• 0ea20e6 postgres: fail on error in pg_heap
• ba8b90b postgres: fill gap alignment in heap tuple
• 1d9ef30 postgres: first correct read of WAL file
• e5f15c5 postgres: fix compilation, fix tests
• bffa008 postgres: fix error in tests
• c23bc42 postgres: fix line endings in error messages, simplify code, add comments
• 9508a20 postgres: fix lint
• 85c0422 postgres: fix linter
• e8bb169 postgres: fix pg_wal when XLogRecord size is more than page size
• 666bbfb postgres: fix some unknown, chanche tests tovalue -> dv
• de3ecf1 postgres: generate docs by embedded md
• 939c7c1 postgres: how_to.md
• dafbf4b postgres: lint fixes
• 6fe5f05 postgres: lint, doc
• 9f5036a postgres: made root an array
• 46e1e33 postgres: make page size const
• 1e24d70 postgres: move SeekAbs(0) to Probe
• ae838b9 postgres: move postgres.md to formats.md, add btree tests
• 03d8fe1 postgres: page sum impl
• 22a6cfa postgres: pg_btree add opaque flags
• dd84d32 postgres: pg_btree begin impl
• b06c9bc postgres: pg_control change default flavour to empty string - it uses versions prober. Fix root name in pg_heap.
• edb5650 postgres: pg_control refactoring
• 9deab2e postgres: pg_heap fix page_begin, page_end
• 7dd7dbe postgres: pg_heap reafactoring
• 00de0a9 postgres: pg_heap refactoring
• 35124bf postgres: pg_heap refactoring
• e8d8cac postgres: pgpro wal implementation
• d7a0f93 postgres: pgpro wal refactoring
• 067f8d5 postgres: pgwal checks
• 7858373 postgres: postgres 10 support
• 5664c0a postgres: refactor ItemIdData
• 08c5352 postgres: refactoring
• 296ce68 postgres: refactoring
• 7b52149 postgres: refactoring
• 7c92715 postgres: refactoring
• 7dedcba postgres: refactoring
• a4d904e postgres: refactoring
• bedc480 postgres: refactoring
• d7bcca0 postgres: refactoring
• e06fa6e postgres: refactoring
• e57c3b9 postgres: refactoring
• f224ed0 postgres: refactoring
• f6f8d5c postgres: refactoring
• ff4b6fd postgres: refactoring - remove GetHeapD
• 7f7f729 postgres: refactoring, tests
• 12b8697 postgres: regenerate docs
• 8f55e17 postgres: remove SeekAbs(0) where it's not used.
• 6e2e44d postgres: remove arg in pg_btree
• 5eea605 postgres: remove duplicate tests
• 5bb8654 postgres: remove lsn parameter in pg_wal
• 60709e5 postgres: remove pg_wal. Failed to implement.
• e87d5a6 postgres: remove unused code
• 448c369 postgres: try to implement pg_wal
• 586c803 postgres: try to implement wal
• 7a89234 postgres: update doc
• c9350de postgres: use bit stream instead of masks to get flags
• c9b263e postgres: version 15 support
• a4c1c5b postgres: wal const
• e311434 postgres: wal decoding implement
• c105fcd postgres: wal impl
• 7c1dfbd postgres: wal implementation
• 26bff14 postgres: wal refactoing
• b09ec2f postgres: wal refactoing
• 069babb postgres: wal refactoring
• bd2bdd6 postgres: wal refactoring
• 015b770 postgres: wal support multiple xlog_body for wal record
• c3ef341 postgresql: general logic for pg_heap, pg_btree
• 721c1ab psotgres: refactoring
• 0327411 ref
• c8ece64 show mock_authentication_nonce as hex
• af0e220 unix time mapper for postgres