Releases · wader/fq

14 Apr 10:55

wader

v0.11.0

d30755b

v0.11.0 Latest

Latest

New iNES/NES 2.0 ROM decoder (thanks @mlofjard) and basic JPEG 2000 format support. jq language improvements and fixes from gojq. And as always various decoder improvements and fixes.

Changes

Add string_truncate option to configure how to truncate long strings when displaying a decode value tree. dd, dv etc set truncate length to zero to not truncate. #919
gojq updates from upstream:
- Implement ltrim, rtrim, and trim functions
- Fix object construction with duplicate keys ({x:0,y:1} | {a:.x,a:.y})
- Fix halt and halt_error functions to stop the command execution immediately
- Fix variable scope of binding syntax ("a" as $v | def f: $v; "b" as $v | f)
- Fix ltrimstr and rtrimstr functions to emit error on non-string input
- Fix nearbyint and rint functions to round ties to even
- Improve parser to allow reduce, foreach, if, try-catch syntax as object values
- Remove pow10 in favor of exp10, define scalbn and scalbln by ldexp
Fix issue using decode value with ltrimstr/rtrimstr.

Format changes

fit
- Skip array fields on pre read messages. #878
- Fixed subfield referencing fields below self in message. #877
jp2c New JPEG 2000 codestream decoder. #928
icc_profile Strip whitespace in header and tag strings. #912
mp4
- Add jp2c, jp2h, ihdr jP JPEG 2000 related boxes support. #928
- Add thmb box support. #897
- Turns out for qt brand hdlr component name might be zero bytes. #896
nes New iNES/NES 2.0 ROM decoder (thanks @mlofjard). #893

Changelog

f7b067b Fixed subfield referencing fields below self in message
9aa99b4 Update docker-golang to 1.22.1 from 1.22.0
0afb5b5 Update docker-golang to 1.22.2 from 1.22.1
2657988 Update github-go-version to 1.22.1 from 1.22.0
33c9391 Update github-go-version to 1.22.2 from 1.22.1
a577c39 Update github-golangci-lint to 1.56.2 from 1.56.1
82d96cf Update github-golangci-lint to 1.57.0 from 1.56.2
72b4569 Update github-golangci-lint to 1.57.1 from 1.57.0
14aeab0 Update github-golangci-lint to 1.57.2 from 1.57.1
735256b Update gomod-golang-x-crypto to 0.20.0 from 0.19.0
043f067 Update gomod-golang-x-crypto to 0.21.0 from 0.20.0
15a7060 Update gomod-golang-x-crypto to 0.22.0 from 0.21.0
85f60df Update gomod-golang-x-net to 0.22.0 from 0.21.0
77c000e Update gomod-golang-x-net to 0.23.0 from 0.22.0
daba6b5 Update gomod-golang-x-net to 0.24.0 from 0.23.0
ba9ecb5 Update gomod-golang-x-term to 0.18.0 from 0.17.0
b2aa59f Update gomod-golang-x-term to 0.19.0 from 0.18.0
1c24f64 Update make-golangci-lint to 1.56.2 from 1.56.1
94e8086 Update make-golangci-lint to 1.57.0 from 1.56.2
4f55b6a Update make-golangci-lint to 1.57.1 from 1.57.0
a3b63b1 Update make-golangci-lint to 1.57.2 from 1.57.1
208b3e6 chore: fix function name in comment
621d7f2 decode: Align some heavily used structs to save space
ee2ee24 decode: Cleanup io panic a bit
e741ca7 doc: Add ImHex to related tools
36e8287 doc: Regenerate after nes and new ansisvg
225fd50 fit: Skip array fields on pre read messages
7500a8b fq: Sort formats
bf7fa07 fq: Use go 1.20 and cleanup
e267040 gojq: Update fq fork
f5fd587 gojq: Update fq fork
ed68511 icc_profile: Strip whitespace in header and tag strings
c8f9cdc interp: Add string_truncate option
0db671f interp: Add todo test for eval error in path
ebffb3b jp2c: Add jpeg2000 codestream format
79992b3 jp2c: Fail probe faster
63f7d79 jp2c: Support probe
b542ff1 lint: More linters and some fixes
c6165c0 mod: go get non-bump tracked modules
1784c43 mp4,avi: Trim spaces for type
2ea70c4 mp4: Add thmb box support
4f90a2e mp4: Decode jP box
70b1b0d mp4: Decode uinf box
87b6c4d mp4: Fix jp2 test
8009b6f mp4: JPEG200 boxes jp2h and ihdr
ed3a126 mp4: Turns out for qt brand hdlr component name might be zero bytes
f3b5404 nes: Add support for iNES/NES 2.0 ROM files
80bccc9 opus,vorbis: More sym snake_case
08df7f4 pkg/cli/test_exp.sh: use command -v
8705273 pssh_playready: Use snake_case sym values
aaa43df test: Replace pmezard/go-difflib with go's internal diff

Contributors

mlofjard

Assets 9

11 Feb 16:17

wader

v0.10.0

8ebe5da

v0.10.0

Adds support for various LevelDB formats (thanks @mikez) and Garmin Flexible and Interoperable Data Transfer format (FIT) (thanks @mlofjard).

And as usual some small fixes and dependency updates.

Changes

On macOS fq now reads init script from ~/.config/fq in addition to ~/Library/Application Support/fq. #871
Switch readline module from own fork to https://github.com/ergochat/readline #854
Updated gojq fork. Notable changes from upstream below. #844
- Fix pre-defined variables to be available in initial modules
- Fix object construction with duplicate keys

Format changes

aac_frame Decode instance tag and common window flag. #859

fit Add support for Garmin Flexible and Interoperable Data Transfer decoder. Thanks @mlofjard #874

This format is used by various GPS tracking devices to record location, speed etc.
Example of converting position information to KML:

# to_kml.jq
# convert locations in a fit structure to KML
def to_kml:
  ( [ .data_records[].data_message as {position_lat: $lat, position_long: $long}
    | select($lat and $long)
    | [$long, $lat, 0]
    | join(",")
    ]
  | join(". ")
  | { kml: {"@xmlns":"http://earth.google.com/kml/2.0"
    , Document: {Placemark: {LineString: {coordinates: .}}}}
    }
  | to_xml({indent:2})
  );

Usage:

# -L to add current directory to library path
# -r for raw string output
# 'include "to_ml";' to include to_kml.jq
# to_kml calls to_kml function
$ fq -L . -r 'include "to_kml"; to_kml' file.fit > file.kml

hevc_sps Fix some incorrect profile_tier_level decoding. #829
html Fix issue parsing elements including SOLIDUS "/". #870
- Upstream issue golang/go#63402
mpeg_es Support ES_ID_Inc and decode descriptors for IOD tags
leveldb_descriptor, leveldb_log, leveldb_table Add support for LevelDB. Thanks @mikez #824
- This format is used by many database backends and applications like Google chrome.
pcapng Decode all section headers instead of just the first. #860
png Fix incorrect decoding of type flags. #847
hevc_sps Fix incorrect decoding of profile_tier_level. #829
tls Fix field name typos. #839
mp4
- Don't try decode samples for a track that has an external reference. #834
- Use box structure instead of track id to keep track for sample table data. #833
- ctts box v0 sample offset seems to be signed in practice but not in spec. #832
webp Decode width, height and flags for lossless WebP. #857

Changelog

6c3914a Add DFDL acronym to README.md
43c7693 Update docker-golang to 1.21.5 from 1.21.4
836af78 Update docker-golang to 1.21.6 from 1.21.5
f9e1e75 Update docker-golang to 1.22.0 from 1.21.6
54da5ef Update ergochat-readline to 81f0f2329ad3 from cca60bf24c9f
bed912c Update github-go-version to 1.21.5 from 1.21.4
3f48e3b Update github-go-version to 1.21.6 from 1.21.5
5bfb2bb Update github-go-version to 1.22.0 from 1.21.6
888b0be Update github-golangci-lint to 1.56.0 from 1.55.2
283380b Update github-golangci-lint to 1.56.1 from 1.56.0
be79c19 Update gomod-golang-x-crypto to 0.16.0 from 0.15.0
6aba928 Update gomod-golang-x-crypto to 0.17.0 from 0.16.0
561fed9 Update gomod-golang-x-crypto to 0.18.0 from 0.17.0
b5388ea Update gomod-golang-x-crypto to 0.19.0 from 0.18.0
0200a4e Update gomod-golang-x-net to 0.19.0 from 0.18.0
2e22e1b Update gomod-golang-x-net to 0.20.0 from 0.19.0
40bcb19 Update gomod-golang-x-net to 0.21.0 from 0.20.0
2a1d970 Update gomod-golang-x-term to 0.16.0 from 0.15.0
310c78e Update gomod-golang-x-term to 0.17.0 from 0.16.0
536583c Update make-golangci-lint to 1.56.0 from 1.55.2
5369576 Update make-golangci-lint to 1.56.1 from 1.56.0
e51c746 aac_frame: CPE: Decode instance tag and common window flag
f5f8e93 bson: Fix jq style a bit
d0a1b30 bump readline to 0.1.0-rc1
b56258c bump,readline: Add bump config and update to latest master
0070e38 bump,readline: Skip bump until not rc
63e0aa3 doc: Fix weird wording in README.md
e5fd1eb doc: Generate svgs with new ansisvg
55470de doc: Update docs to include fit format
c47756d doc: Update svgs after ansisvg update
54c6f0c fit: Added support for "invalid" value checking.
46dbf5b fit: Added support for ANT+ FIT format (used by Garmin devices)
6219d57 fit: Added support for dynamic subfields
33e5851 fit: Fix field casing to snake_case. Misc cleanup.
76307e4 fit: Formatted date/time description for timestamp fields
8862280 fit: Made long/lat present as float64
788088f fit: Show crc as hex, lower case "invalid" and some style harmonization
a07ce11 fq: Relicense float80 code to MIT
5829c6b gojq: Update fq fork
d155c80 gojq: Update fq fork
a793d10 goreleaser: Fix deprecated options
919e079 hevc_sps: Fix some incorrect profile_tier_level decoding
69ae765 interp: Fix crash when using line_bytes=0
2176600 interp: Support ~/.config/fq as fallback on macOS
fb910bd ldb: first draft
efc59a8 ldb: uncompression support
41f27a1 leveldb: add torepr for descriptor
2df0f0f leveldb: add log and descriptor decoders
b05aa99 leveldb: address PR comments
2f5f183 leveldb: decode unfragmented .log files further; fix UTF8 decoding
e826f09 leveldb: fix Errorf arguments
4283091 leveldb: fix all.fqtest failures
287ed36 leveldb: fix metaindex keys, refactoring, and jq syntax per PR
8665df5 leveldb: fix table's data blocks' internal keys decoding
08e3d2d leveldb: improve stringify by preallocating result
3a396e1 leveldb: improve log documentation
1ba8dec leveldb: in some properties, change spaces to underscores
e735cea leveldb: propagate error
07ad940 leveldb: rename "suffix" to "sequence_number_suffix"
78a3e94 leveldb: rename functions and add comments
cc0d5a8 leveldb: update docs
fe1099b leveldb: updates per PR comments
0d06e0a mp4: Don't decode samples if track has external data reference
d2c5ce5 mp4: Use box structure instead of track id to keep track for sample table data
aadf26f mp4: ctts v0 sample_offset seems to be signed in practice
fca55b2 mpeg_es: Support ES_ID_Inc and decode descriptors for IOD tags
e3af467 pcapng: Decode all section headers, clenaup descriptions
38b4412 png: Type flags were off-by-one bit
fb1c625 readline,bump: Fix version compare link
41226f4 readline: Switch to ergochat/readline
cd572d3 readline: Update to 0.1.0 and add bump config
7906f33 test: Support to more common -update flag
3b7cc1f tls: Fix field name typos
b0421df webp: Decode width, height and flags for lossless webp

Contributors

mikez and mlofjard

Assets 9

25 Nov 17:53

wader

v0.9.0

1482145

v0.9.0

Changes

Make synthetic values not look like decode values with a zero range. #777
Bit ranges are now displayed using exclusive end notation to be more consistent. For example if a field at byte offset 5 is 8 bit in size using the <byte>[.<bits>] notation it is now shown as 5-6, before it was shown as 5-5.7. See usage documentation for more examples. #789
Improve colors when using a light background. Thanks @adedomin for reporting. #781
Better from_jq error handling. Thanks @emanuele6 for reporting. #788

Updated gojq fork. Notable changes from upstream below. #808

Adds pick/1 function. Outputs value and nested values for given input expression.

# pick out all string values from a ELF header
$ fq '.header | pick(.. | select(type == "string"))' /bin/ls
{
  "ident": {
    "data": "little_endian",
    "magic": "\u007fELF",
    "os_abi": "sysv",
    "pad": "\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
  },
  "machine": "x86_64",
  "type": "dyn"
}
# bonus tip:
# if you only need to pick one level then jq has a shortcut syntax
$ fq '.header | {type, machine}' /bin/ls
{
  "machine": "x86_64",
  "type": "dyn"
}

Implements comment continuation with backslash.

Updated gopacket to 1.2.1. Notable changes from upstream below. #815
- fix(ip4defrag): allow final fragment to be less than 8 octets by @niklaskb
- refactor: don't fill empty metadata slots by @smira
- refactor: optimize port map to switch statement by @smira

Decoder changes

avi

Some general clean up fixes. Thanks to Marion Jaks at mediathek.at for testing and motivation.
Add extended chunks support and decode_extended_chunks option. This is trailing chunks used by big AVI files. #786

Add type, handler, compression (video) and format_tag (audio) per stream. #775

$ fq '.streams[0]' file.avi
      │00 01 02 03 04 05 06 07│01234567│.streams[0]{}: stream
0x1680│      00 00 00 01 67 f4│  ....g.│  samples[0:3]:
0x1688│00 0d 91 9b 28 28 3f 60│....((?`│
0x1690│22 00 00 03 00 02 00 00│".......│
*     │until 0x2409.7 (3464)  │        │
      │                       │        │  type: "vids"
      │                       │        │  handler: "H264"
      │                       │        │  compression: "H264"

Properly use sample size field when decoding samples. #776

exif (and tiff)
- Handle broken last next ifd offset by treating it as end marker. #804
gzip
- Correctly handle multiple members. Thanks @TomiBelan for the bug report and assistance. #795
- Now gzip is modelled as a struct with a members array and a uncompressed field that is the concatenation of the uncompressed members.
macho
- Properly respect endian when decoding some flag fields. #796
- Move formatted timestamp to description so that numeric value is easier to access. #797
matroska
- Support decoding EBML date type. #787
protobuf
- No need to use synthetic fields for string and bytes. #800
webp
- Refactor to use common RIFF decoder and also decode VP8X, EXIF, ICCP and XMP chunks. #803

zip Better timestamp support and fixes

Fix incorrect MSDOS time/date decoding and add extended timestamp support. Also remodel time/date to be a struct with raw values, components and a synthetics UTC unixtime guess. Thanks @TomiBelan for the bug report and assistance. #793

$ fq '.local_files[] | select(.file_name == "a").last_modification' file.zip
    │00 01 02 03 04 05 06 07│01234567│.local_files[3].last_modification{}:
0xd0│            81 01      │    ..  │  fat_time: 0x181
    │                       │        │  second: 2 (1)
    │                       │        │  minute: 12
    │                       │        │  hour: 0
0xd0│                  73 53│      sS│  fat_date: 0x5373
    │                       │        │  day: 19
    │                       │        │  month: 11
    │                       │        │  year: 2021 (41)
    │                       │        │  unix_guess: 1637280722 (2021-11-19T00:12:02)

Changelog

b702218 Update docker-golang to 1.21.2 from 1.21.1
d704711 Update docker-golang to 1.21.3 from 1.21.2
c31fc87 Update docker-golang to 1.21.4 from 1.21.3
861487d Update github-go-version to 1.21.2 from 1.21.1
d766356 Update github-go-version to 1.21.3 from 1.21.2
caef93c Update github-go-version to 1.21.4 from 1.21.3
de7fdae Update github-golangci-lint to 1.55.0 from 1.54.2
60edf97 Update github-golangci-lint to 1.55.1 from 1.55.0
534a2c8 Update github-golangci-lint to 1.55.2 from 1.55.1
906bc3b Update gomod-golang-x-crypto to 0.14.0 from 0.13.0
8d4d18d Update gomod-golang-x-crypto to 0.15.0 from 0.14.0
f108194 Update gomod-golang-x-net to 0.16.0 from 0.15.0
5381f38 Update gomod-golang-x-net to 0.17.0 from 0.16.0
14fe728 Update gomod-golang-x-net to 0.18.0 from 0.17.0
1011f19 Update gomod-golang/text to 0.14.0 from 0.13.0
527aad6 Update gomod-gopacket to 1.2.0 from 1.1.1
0c22c79 Update make-golangci-lint to 1.55.0 from 1.54.2
5f06364 Update make-golangci-lint to 1.55.1 from 1.55.0
36576a5 Update make-golangci-lint to 1.55.2 from 1.55.1
d703321 avi: Add extended chunks support and option
55521bb avi: Add stream type constants
5196554 avi: Add type, handler, format_tag and compreession per stream
0f225c3 avi: Add unused field for extra indx chunk space
df085b9 avi: Handle stream sample size
c7ec18d avi: Increase sample size heuristics to 32bit stereo
a745b12 avi: More correct strf chunk extra data
9b10e59 avi: Only use sample size heuristics if there is no format
23ae4d9 decode,interp: Make synthetic values more visible and not act as decode values
5abf151 doc: Remove spurious backtick
02b3527 exif,tiff: Handle broken last next ifd offset by treating it as end marker
dc376f3 gojq: Update rebased fq fork
ac276ee gzip: Correctly handle multiple members
45a8dd9 interp: Better from_jq error handling
051a70b interp: Change bit ranges to use exclusive end
29e7541 interp: Fix infinite recursion when casting synthetic raw value into a jq value
c28163f interp: Improve colors when using light background
797c7d9 macho: Move timestamp string to description
71a5fc9 macho: Respect endian when decoding flags
1d14ea5 matroska: Decode ebml date type
b24ed16 mod: Update golang.org/x/exp and github.com/gomarkdown/markdown
5e2e49e protobuf: No need for synthetic for string and bytes value
6034c70 webp,avi,wav,aiff: Trim RIFF id string
9e58067 webp: Refactor to use riff code and decode VP8X, EXIF, ICCP and XMP chunks
a83cac6 zip: Fix incorrect time/date, add extended timestamp and refactor

Contributors

smira, TomiBelan, and 3 other contributors

Assets 9

27 Sep 16:52

wader

v0.8.0

3988399

v0.8.0

Fix handling of shadowing order for _<name> keys, 3 new decoders caff, moc3 and opentimestamps, some smaller fixes and improvements.

In other jq news jq 1.7 was finally released 5 years since the last release! also happy to now be part of the jq maintainance team.

Changes

New decoders caff, moc3 and opentimestamps. See below for details.
Fix shadowing of underscore prefixed keys (_<name>) for text formats like json, yaml etc. #757
This happenned because fq has a bunch of internal underscore prefixed "extra" keys that is used for various things and these had priority even when there already existed a "value" key with same name.
```
$ fq -n '`{"_format": 123}` | fromjson | ._format'
```
Now 123, before "json".
```
$ fq -n '`{}` | fromjson | ._missing'
```
Now null, before error
Rename --null/nul-output to --raw-output0 and also clarify that NUL and new lines are outputted after and not between each output.
This is to be in sync with jq (jqlang/jq#2684). #736
Updated gojq fork with fixes from upstream:
- Improved error messages for indices, setpath, delpaths
- Add abs function
- Change behavior of walk with multiple outputs
- Change zero division to produce an error when dividend is zero
- Fix empty string repeating with the maximum integer
- Fix string multiplication by zero to emit empty string
- Remove deprecated leaf_paths function
Fix split in combination with binary to not include separator. #767

Decoder changes

caff Add archive format decoder. Thanks @Ronsor #747
- CAFF is an archive format usually found with the extensions .cmo3 and .can3 used by Live2D Cubism.
id3v2 Handle W000-WZZZ and W00-WZZ URL frames. #758
matroska Update spec and regenerate. #737
moc3 Add Live2D Cubism MOC3 decoder. Thanks @Ronsor #747
- MOC3 is a format for 2D rigged puppets, somewhat like Flash.
mp3_frame_xing Detect lame ext more similar to ffmpeg and mediainfo. #763
mp4
- Decode sgpd (Sample group definition box) entries. Thanks Sergei Kuzmin @ksa-real #707
- Decode cslg (Composition to decode timeline mapping) box. #754
- Decode emsg (Event message) and id3v2 message data. #755
- Nicer trimmed major brand for ftyp. #723
opentimestamps Add OpenTimestamps decoder. Thanks @fiatjaf #769

Changelog

4031082 Update docker-golang to 1.20.6 from 1.20.5
6daa0aa Update docker-golang to 1.20.7 from 1.20.6
8bd7b6d Update docker-golang to 1.21.0 from 1.20.7
bff668c Update docker-golang to 1.21.1 from 1.21.0
8e705aa Update github-go-version to 1.20.6 from 1.20.5
3828b42 Update github-go-version to 1.20.7 from 1.20.6
c09d50a Update github-go-version to 1.21.0 from 1.20.7
30b27a5 Update github-go-version to 1.21.1 from 1.21.0
104c3bd Update github-golangci-lint to 1.54.0 from 1.53.3
7906a46 Update github-golangci-lint to 1.54.1 from 1.54.0
31de3f9 Update github-golangci-lint to 1.54.2 from 1.54.1
8394729 Update gomod-golang-x-crypto to 0.12.0 from 0.11.0
ebb71e2 Update gomod-golang-x-crypto to 0.13.0 from 0.12.0
c8aae66 Update gomod-golang-x-net to 0.13.0 from 0.12.0
a46ee65 Update gomod-golang-x-net to 0.14.0 from 0.13.0
07069a5 Update gomod-golang-x-net to 0.15.0 from 0.14.0
79432e7 Update gomod-golang/text to 0.12.0 from 0.11.0
2f8ebf1 Update gomod-golang/text to 0.13.0 from 0.12.0
1fa14a0 Update make-golangci-lint to 1.54.0 from 1.53.3
fc4101d Update make-golangci-lint to 1.54.1 from 1.54.0
4e20e04 Update make-golangci-lint to 1.54.2 from 1.54.1
013cc2f caff: eliminate gaps and specify unused fields
6a3fecd caff: include uncompressed bits for proper decompressed entries that can't be decoded as a format
da41a8d caff: initial implementation
23e660f caff: minor formatting changes
fa11572 caff: obfuscation key is a signed integer, add test data
29084e3 caff: remove dead code
4dd0f6d caff: run go fmt
b3759de caff: run go fmt
cc58c4b caff: update doc/formats.md
d5345f0 cli: Rename --null/nul-output to --raw-output0
c0352f2 decode,interp: Don't shadow _key and error on missing _key
44f0060 dev,jq: Reformat jq code to look more the same
9cd1d0f dev: Move examples and snippets to wiki
f15f9bc doc,moc3,caff: Add author and regenerate docs
406f392 doc: Move up and update differences jq section a bit
8edef78 docker: Change to bookworm
56fec2a elf: Fix broken static and segfault tests
fa3dba1 gojq: Update fq fork
0cefc46 golangci: Fix gosec aliasing warnings
0d01476 gomod: Update x/exp and gomarkdown
c503bc1 html: Add forgotten re test
0efe5a2 id3v2: Handle W000-WZZZ,W00-WZZ URL frames
a614c9d interp: split: Correctly split binary
3af0462 luajit: file null terminator: raw bits, validate
c07673a matroska: Update spec and regenerate
441fcd0 moc3, caff: update tests and README
f7eb027 moc3: Fix field order in blend_shape_keyform_bindings structure, version detection in count_info
03ba71b moc3: add support for version 5
d3073c6 moc3: add test data for new version 5
ce40fd1 moc3: consistency - scales array contains value elements, not scale elements
fac1e68 moc3: count_info: extra space is reserved, not normal alignment/padding
e424e29 moc3: eliminate gaps and properly handle padding, fix version 5 format decoding
092662e moc3: initial implementation
3caf34e moc3: nicer tree structure, use more meaningful names for array elements
20f02e7 moc3: remove dead code
6d10a25 moc3: update certain array element names, explicitly mark unused or reserved space
833b063 moc3: update test data
14f233d moc3: update tests
c4e8644 mod: Update golang.org/x/exp and github.com/gomarkdown/markdown
0699c80 mp3_frame_xing: Detect lame ext more similar to ffmpeg and mediainfo
e50028a mp4,mpeg_es: Decode iods box and MP4_IOD_Tag OD
312d807 mp4: Decode cslg box
bedd719 mp4: Decode emsg box
97194ad mp4: Nicer major brand and handle some qt brand short strings better
cc8e6f1 opentimestamps: abstract away file digest sizes and support sha1, ripemd160 and keccac256.
64a4ff2 opentimestamps: account for unknown attestation types.
912f411 opentimestamps: add help text.
cef5faa opentimestamps: add parser.
1aa557d opentimestamps: add tests.
5e7c01a opentimestamps: address comments and improve things.
976a756 opentimestamps: one last make doc.
0a22a32 opentimestamps: satisfy linter.
456a6a4 protobuf_widevine: Make protection_scheme constants less magic

Contributors

fiatjaf, Ronsor, and ksa-real

Assets 9

07 Jul 09:44

wader

v0.7.0

d15a41f

v0.7.0

Added LuaJIT bytecode decoder by @dlatchx, otherwise mostly small things. Been busy with nice weather and helping our getting jq development and maintenance back on track.

Changes

Better performance of binary arrays when they only include 0-255 numbers or strings. #704

Make tovalue on binary, in addition decode value binary, respect bits_format options. #677

# uses -V to do tovalue to get a hex string
# uses -r to output "raw" string
$ fq -o bits_format=hex -Vr -n '[1,0xff,3] | tobytes'
01ff03

Updated gojq fork with fixes from upstream: #679
- Improved error messages
- @urid URI format function

Decoder changes

luajit Add LuaJIT bytecode decoder. Thanks @dlatchx #709
mp4 Improved sample group definition box sgpd entries decoder. Thanks @ksa-real #707
mp4 Improved user metadata udta structure decoding #676
wav Decode bext chunk. #712

Changelog

47b9060 Improved README.md
d02b70f Update README.md
64e17f0 Update docker-golang to 1.20.5 from 1.20.4
6faed67 Update github-go-version to 1.20.5 from 1.20.4
b9fce9b Update github-golangci-lint to 1.53.1 from 1.52.2
ff4048c Update github-golangci-lint to 1.53.2 from 1.53.1
76e0e17 Update github-golangci-lint to 1.53.3 from 1.53.2
8e75dc9 Update gomod-BurntSushi/toml to 1.3.2 from 1.2.1
6dc0746 Update gomod-golang-x-crypto to 0.10.0 from 0.9.0
98351ff Update gomod-golang-x-crypto to 0.11.0 from 0.10.0
939d98c Update gomod-golang-x-net to 0.11.0 from 0.10.0
660ca03 Update gomod-golang-x-net to 0.12.0 from 0.11.0
36ef2a2 Update gomod-golang/text to 0.10.0 from 0.9.0
0eb6557 Update gomod-golang/text to 0.11.0 from 0.10.0
a079b73 Update gomod-gopacket to 1.1.1 from 1.1.0
c3e104b Update make-golangci-lint to 1.53.1 from 1.52.2
7c1da0e Update make-golangci-lint to 1.53.2 from 1.53.1
47ea6cf Update make-golangci-lint to 1.53.3 from 1.53.2
fd2cb6f doc: Fix broken link in README
db2e621 go fmt
38cb829 gojq: Update rebased fq fork
41f40b7 interp: Add to binary fast path for arrays with only 0-255 numbers and strings
b2c0e5f interp: Make binary also respect bits_format
b24063b luajit: *.fqtest: add comments for generating .luac from source
bdf158b luajit: add luajit.md
93c9696 luajit: add to probe group
32300a3 luajit: check binary.Read() error
a83576a luajit: clarify description
751ee5a luajit: explain LuaJITDecodeKNum, fix negative in bug
3561c08 luajit: fallbackUintMapSymStr
5d9a08c luajit: fix regression: (u64 vs i64)
64c11be luajit: improve debuginfo decoding
1afdf8b luajit: initial support
29ab66b luajit: lowercase flags
e44f5c0 luajit: magic number: raw bits, check with assert
23b9eea luajit: make doc
715f850 luajit: opcodes: implement scalar.UintMapper
c3a123a luajit: remove unecessary dependency
64c92da luajit: remove unused variable
52ce818 luajit: split in smaller decode functions
441d246 luajit: standardize field names (key/value/type ect.)
eb819dd luajit: tests: improve coverage
dd594f4 luajit: tests: rename lua source file
c42fb9e luajit: typo
08ae661 luajit: use UTF8 strings
1da8069 mp4: udta: Improve length/lang box probe and support empty value
e869d8a sgpd box entries parsing
8c75509 wav: Decode bext chunk

Contributors

dlatchx and ksa-real

Assets 9

16 May 14:06

wader

v0.6.0

4335c30

v0.6.0

Adds decoders for PostgreSQL btree, control and heap files. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka

Adds new option skip gaps and output binary as hex string.

Make bits/bytes formats work a bit more intuitive.

Bug fixes to to_hex, to_base64 and trim functions.

Bug fixes and additions to bson, bitcoin_transaction, mp4, id3v2, html, and matroska formats.

Changes

bits,bytes now are real binaries and not raw decode value. This means they behave more like you would expect. #666

# build your own strings(1)-like tool:
# scan matches range in a binary using a regexp and output ranges as new binaries
# \w\s looks for whitespace and alpha/numeric characters
# uses `...` raw string literal to not have to escape
# select/test to only include strings containing "thread"
# dd to display with no truncation
$ fq -d bytes 'scan(`[\w\s]{16,}`) | select(test("thread")) | dd' file.mp4
     │00 01 02 03 04 05 06 07│01234567│
0x250│36 20 6c 6f 6f 6b 61 68│6 lookah│.: raw bits 0x250-0x262.7 (19)
0x258│65 61 64 5f 74 68 72 65│ead_thre│
0x260│61 64 73               │ads     │
     │00 01 02 03 04 05 06 07│01234567│
0x260│            31 20 73 6c│    1 sl│.: raw bits 0x264-0x273.7 (16)
0x268│69 63 65 64 5f 74 68 72│iced_thr│
0x270│65 61 64 73            │eads    │

to_hex,to_base64 now correctly handles raw decode values, before the raw bits would be turned into codepoints and then binary UTF-8 possibly introducing invalid codepoints (0xfffd). Thanks @Rogach #672
```
$ fq -r '.uncompressed | to_hex' file.gz
f6f2074cf77d449d

# with the change to add hex bits format you can also do this now
$ fq -Vr -o bits_format=hex .uncompressed file.gz
f6f2074cf77d449d
```
tovalue Now output a "deep" jq value, before it was shallowly a jq value which could be confusing, ex tovalue | .header could be a decode value.
New option skip_gaps for -V/tovalue can be used to filter out gap fields when represented as JSON. Gaps are bit ranges that no decoder added any field for. #649
- Can for example be used to remove trailing data that will end up as gap0 etc.
- Note that gaps in an array that gets filtered out might affect index numbers.

Add bits_format=hex to represent raw bits as hex string in JSON. #673

# output decode tree, JSON with binaries as strings and JSON with binaries as hex strings
$ fq '.packets[0].packet | ., tovalue, tovalue({bits_format:"hex"})' file.pcap
    │00 01 02 03 04 05 06 07│01234567│.packets[0].packet{}: (ether8023_frame)
0x28│8c 85 90 74 b8 3b      │...t.;  │  destination: "8c:85:90:74:b8:3b" (0x8c859074b83b)
0x28│                  e8 de│      ..│  source: "e8:de:27:c8:9a:6e" (0xe8de27c89a6e)
0x30│27 c8 9a 6e            │'..n    │
0x30│            08 06      │    ..  │  ether_type: "arp" (0x806) (Address Resolution Protocol)
0x30│                  00 01│      ..│  payload: raw bits
0x38│08 00 06 04 00 01 e8 de│........│
0x40│27 c8 9a 6e c0 a8 01 01│'..n....│
0x48│00 00 00 00 00 00 c0 a8│........│
0x50│01 e6                  │..      │
{
  "destination": "8c:85:90:74:b8:3b",
  "ether_type": "arp",
  "payload": "\u0000\u0001\b\u0000\u0006\u0004\u0000\u0001\ufffd\ufffd'Țn\ufffd\ufffd\u0001\u0001\u0000\u0000\u0000\u0000\u0000\u0000\ufffd\ufffd\u0001\ufffd",
  "source": "e8:de:27:c8:9a:6e"
}
{
  "destination": "8c:85:90:74:b8:3b",
  "ether_type": "arp",
  "payload": "0001080006040001e8de27c89a6ec0a80101000000000000c0a801e6",
  "source": "e8:de:27:c8:9a:6e"
}

Explicit use of d/display on a binary will now always hexdump, this feels more intuitive. Before it could output raw binary as display is used as an implicit output function. Implicit (you don't mention d at all) can still output raw binary. #665

# outputs raw binary (if stdout is not a tty)
$ fq -n '[1,2,3] | tobytes' | xxd
00000000: 0102 03

# outputs hexdump (even if stdout is not a tty) as we explicitly use d
$ fq -n '[1,2,3] | tobytes | d' | cat
   |00 01 02 03 04 05 06 07|01234567|
0x0|01 02 03|              |...|    |.: raw bits 0x0-0x2.7 (3)

trim can now handle multi-line strings. #668
Help texts originated from markdown are now rendered a bit more nicely and compact. #661
Fix weirdly rendered monospace font in demo SVG when using Japanese locale and Chrome on macOS. Thanks @acevif helping with investigation. #655 #662
Fix a bunch of typos. Thanks @kianmeng and @castilma. #644 #670

Decoder changes

bits,bytes Is a proper binary not a raw decode value. #666
bitcoin_transaction Properly decode witness items array. #671 Thanks @Rogach
bson Add javascript, decimal128, minkey and maxkey support. Fix decoding of datetime and use the correct size type for binary and document size. Thanks Matt Dale @matthewdale. #650
id3v2
- Add WXXX frame support. #656
- Add CTOC flags support. #657
html Is now probeable. #667
matroska Fallback raw is probe fail or file_data #645
mp4
- Better description for terminator atom (is a QuickTime thing) #651
- ctts,infe,iinf,trun more proper decoding based on version. #643
- Use correct QuickTime epoch for timestamps. #674
pg_btree,pg_control,pg_heap Add PostgreSQL btree, control and heap support. Thanks Pavel Safonov @pnsafonov and Michael Zhilin @mizhka. #415

Changelog

d4f8dfa Add flavour arg to postgres parser
3c6ea87 Add heap infomask flags parser to PostgreSQL
0107d12 Add pgproee14 flavour to postgres
9a96da8 Add postgres pg_control parser
3b81d99 DBState enum for postgres
51878dc PostgreSQL heap page parser implememtation.
6ed0263 PostgreSQL: accept only normal item pointers
ffc08cf PostgreSQL: add heap for pgpro14
b4ae1d5 PostgreSQL: add pg_control for pgproee 12
a653710 PostgreSQL: add pg_control to pgpro14
d9de2d4 PostgreSQL: fix
96a86e2 PostgreSQL: fixes
96a96a5 PostgreSQL: heap impl for pgproee 12
6618e76 PostgreSQL: heap impl for version 11
ce9ae76 PostgreSQL: heap tuples implementation
ffd7c9b PostgreSQL: implement pgproee 14
850dc60 PostgreSQL: lp_flags format
f5278f3 PostgreSQL: pg_control for ver 12
07056c2 PostgreSQL: pg_control impl for pgproee 10
972c5a3 PostgreSQL: pg_control, pgheap impl for pgproee13
d148727 PostgreSQL: pgheap impl for pgproee10
2d3884a PostgreSQL: pgproee11 heap impl
d8b891c PostgreSQL: pgproee11 pg_control impl
b722b21 PostgreSQL: ref
fdb3b3e PostgreSQl: fix offset
621c4c4 PostgreSQl: heap impl for version 13
c273a6c PostgreSQl: pg_control impl for version 13
01b380e PostgrreSQl heap decode refactoring
bebdfa9 Try to implement pgwal - fail.
c9d3b8f Update docker-golang to 1.20.4 from 1.20.3
2a5927b Update github-go-version to 1.20.4 from 1.20.3
c405afd Update gomod-golang-x-crypto to 0.9.0 from 0.8.0
ed0cd6d Update gomod-golang-x-net to 0.10.0 from 0.9.0
deaf5ef WalLevel for postgres
f069ddc [WIP] initial attempt to add postgres
2b1bdfb add icu version mapper
cf1e7b2 add pgpro11 for postgres
9570d4d add pgpro12 postgres
7bf6b11 add pgpro13 heap
2b3035f add pgpro13 to postgres
f56c72d add postgres tests for mem, cpu profiling
2ee01f7 allow to change FillGaps in decoder
a3361e7 bitcoin: fix witness item structs
222cd88 bits,bytes: Behave as binary instead of raw decode value
9a982d0 bson: add BSON test file generator module and correct BSON format docs
40630d3 bson: fix doc formatting and add author info
2017ff8 bson: support all non-deprecated types and fix int/uint bugs
b08ef00 decode,interp: Refactor format groups into a proper struct
af68511 dev,doc Clarify some dev docs and rename launch.json to be a template
97c952b doc: Add some more examples
88be3a7 doc: Hopefully fix svg fixed font issue
dd4fa26 doc: fix typos
b0e4da2 fix non-ascii characters handling in to_hex and to_base64 functions
a4a332b formats: Clenaup naming a bit
b3b6cd0 gzip.go: fix typo in variablename: delfate
2c505fe help,markdown: Rewrote and made text rendering nicer
e2eb667 html: Add to probe group
d010dce id3v2: Add WXXX (desc/url) frame support
f237db2 id3v2: Decode CTOC flags
684a083 interp,decode: Support decode group argument
8a468f4 interp: Add hex bits format
c512713 interp: Add skip_gaps option for tovalue/-V
033498b interp: Don't output raw binary if display is called explicitly
ee66fec interp: Make tovalue output behave as jq value
d5ae116 interp: trim: Add multi-line support
8d317dd lsn mapper
fcd7fbc mappers for postgres
8941b13 matroska: file_data: Fallback to raw if probe fails
7adc1e7 mp4: Better description for QuickTime terminator atom
493848a mp4: Use correct epoch for quicktime timestamps
3c6d31b mp4: ctts,infe,iinf,trun: More ISOMFF version handling
d6f785c pcap: Add forgotten help test
8cf83fb pg_control implementation
441ecee pgpro version mapper
5ff6273 postgres: add additional checks in pg_heap
4676590 postgres: add argument to calc page's check sum correctly
7fd4109 postgres: add btree index tests
1aa08e3 postgres: add btree, pg_control to how_to.md
2423f86 postgres: add how_to.md - how to generate test files for postgres
97bbc22 postgres: add page arg in pg_btree, change args names in pg_heap
6aed238 postgres: add pg_wal for pgproee11 as copy of postgres14
08eb303 postgres: add postgres format docs, refactoing postgres flavours
90386a6 postgres: add postgres.md to format
09c42c3 postgres: add state to wal struct
c2591ac postgres: add test data with specific values
de68785 postgres: add test files
4db1284 postgres: add tests
3c7ed5d postgres: add test...

Contributors

mizhka, kianmeng, and 5 other contributors

Assets 9

13 Apr 15:47

wader

v0.5.0

23d980d

v0.5.0

Mostly a bug fix release but adds -V for easy JSON output.

Changes

Add -V argument to default output JSON instead of decode tree in case of decode value. #385 Thanks @peterwaller-arm for reminding me to merge this.

# default in case of decode value is to show a hexdump tree
$ fq '.headers | grep_by(.id=="TSSE").text' file.mp3
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x0c│                           4c 61 76│         Lav│.headers[0].frames[0].text: "Lavf58.76.100"
0x18│66 35 38 2e 37 36 2e 31 30 30 00   │f58.76.100. │

# with -V an implicit "tovalue" is done
$ fq -V '.headers | grep_by(.id=="TSSE").text' file.mp3
"Lavf58.76.100"

# and in combination with -r will for strings output a "raw string" without quotes
# for other types like number, object, array etc -r makes not difference (same as jq)
$ fq -Vr '.headers | grep_by(.id=="TSSE").text' file.mp3
Lavf58.76.100

As a side note -V can be used with binary type also. Then the binary data will be interpreted as UTF-8 and turned into a string.

# trailing null terminator ends up as codepoint zero `\u0000`
$ fq -V '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3
"Lavf58.76.100\u0000"

# with -r null terminator and a new line is outputted
$ fq -Vr '.headers | grep_by(.id=="TSSE").text | tobytes' file.mp3 | hexdump -C
00000000  4c 61 76 66 35 38 2e 37  36 2e 31 30 30 00 0a     |Lavf58.76.100..|
0000000f

# in contrast raw binary output has no new line separator
$ fq '.headers | grep_by(.id=="TSSE").text | tobytes' doc/file.mp3 | hexdump -C
00000000  4c 61 76 66 35 38 2e 37  36 2e 31 30 30 00        |Lavf58.76.100.|
0000000e

Fix issue using decode value in object passed as argument to internal function. #638

# this used to fail but now works
fq '.tracks[0].samples[10] | avc_au({length_size: <decode value>})' file.mp4

Some typo fixes. Thanks @retokromer and @peterwaller-arm

Decoder changes

aiff Basic AIFF decoder added. #614
matroska Update to latest specification. #640
msgpack Fix bug decoding some fixstr lengths. #636 Thanks @schmee for reporting.

Changelog

4ad1cce Update docker-golang to 1.20.3 from 1.20.2
f7dca47 Update github-go-version to 1.20.3 from 1.20.2
c960893 Update github-golangci-lint to 1.52.0 from 1.51.2
0a6b46c Update github-golangci-lint to 1.52.1 from 1.52.0
c4eb67d Update github-golangci-lint to 1.52.2 from 1.52.1
19140a6 Update gomod-creasty-defaults to 1.7.0 from 1.6.0
6e5df72 Update gomod-golang-x-crypto to 0.8.0 from 0.7.0
6c4aebf Update gomod-golang-x-net to 0.9.0 from 0.8.0
f13cc97 Update gomod-golang/text to 0.9.0 from 0.8.0
e2af57e Update gomod-gopacket to 1.1.0 from 1.0.0
a63fd68 Update make-golangci-lint to 1.52.0 from 1.51.2
d3d1f0e Update make-golangci-lint to 1.52.1 from 1.52.0
f0b0845 Update make-golangci-lint to 1.52.2 from 1.52.1
dc4a82e aiff: Add basic decoder
c5f6809 decode,fuzz,dev: Move recoverable error check to recoverfn.Run
980ecdb decode: Add float 80 reader
a6c4db7 decode: Cleanup old unused help system code
87e5bb1 fix typo
0b6ef2a golangci-lint: Disable revive unused-parameter and update for new default config
427ce78 interp: Add --value-output/-V option to do tovalue before output
9a1ef84 interp: Allow and convert JQValues:s (ex decode value) in function arg objects
3dd2c61 interp: Fix input completion regression in sub-REPLs
5415bfc interp: Make completion work again
2a2b64d matroska: Update ebml specification
82da99c msgpack: Add str, array and object type tests
97360d6 msgpack: fixstr length field is 5 bits
ffc66db readline: remove direct access to (*Instance).Config
e1b0231 wav: Cleanup avi leftovers

Contributors

retokromer, schmee, and peterwaller-arm

Assets 9

08 Mar 16:49

wader

v0.4.0

29bb15a

v0.4.0

TLS decode and decryption, better streaming matroska/webm support, support raw IP in PCAP and bug fixes.

Changes

Fix panic when interrupting big JSON output. #573
Support passing options (-o name=value) to nested decoders. #589
- Allows for example to pass keylog to a TLS decoder inside a PCAP file or to tell a container decoders to not decode samples inside a ZIP file etc.
Exit with error if -o name=@path fails to read file at path. #597

Decoder changes

id3v2 Properly decode CTOC subframes. #606
matroska
- Now supports streaming matroska and webm better (master elements with unknown size). #576 #581
- Add decode_samples option. #574
- Spec update and clean up of symbols and descriptions. #580
pcap,pcapng Support raw IPv4 and IPv6 link frames. #599 #590

tls Add Transport layer security decoder and decryption. #603

Supports TLS 1.0, 1.1, 1.2 and some SSL 3.0.
Decodes records and most messages and extensions.
Can decrypt most common cipher suites if a keylog is provided. See documentation for list of supported ciphers suites.

# show first 50 bytes of decrypted client/server TLS application data stream
# -o keylog=@file.pcap.keylog is used to read keylog from a file
# first .stream is TCP stream, second .stream the application data stream
$ fq -o keylog=@file.pcap.keylog '.tcp_connections[0].["client", "server"].stream.stream | tobytes[0:50] | dd' file.pcap
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x00│47 45 54 20 2f 64 75 6d 70 2f 6c 6f│GET /dump/lo│.: raw bits 0x0-0x31.7 (50)
0x0c│67 20 48 54 54 50 2f 31 2e 31 0d 0a│g HTTP/1.1..│
0x18│48 6f 73 74 3a 20 69 6e 77 61 64 65│Host: inwade│
0x24│72 2e 63 6f 6d 0d 0a 55 73 65 72 2d│r.com..User-│
0x30│41 67                              │Ag          │
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│
0x00│48 54 54 50 2f 31 2e 31 20 32 30 30│HTTP/1.1 200│.: raw bits 0x0-0x31.7 (50)
0x0c│20 4f 4b 0d 0a 41 63 63 65 70 74 2d│ OK..Accept-│
0x18│52 61 6e 67 65 73 3a 20 62 79 74 65│Ranges: byte│
0x24│73 0d 0a 43 6f 6e 74 65 6e 74 2d 4c│s..Content-L│
0x30│65 6e                              │en          │

# show first TLS record from server
$ fq '.tcp_connections[0].server.stream.records[0] | d' file.pcap
    │00 01 02 03 04 05 06 07 08 09 0a 0b│0123456789ab│.tcp_connections[1].server.stream.records[0]{}: record
0x00│16                                 │.           │  type: "handshake" (22) (valid)
0x00│   03 03                           │ ..         │  version: "tls1.2" (0x303) (valid)
0x00│         00 40                     │   .@       │  length: 64
    │                                   │            │  message{}:
0x00│               02                  │     .      │    type: "server_hello" (2)
0x00│                  00 00 3c         │      ..<   │    length: 60
0x00│                           03 03   │         .. │    version: "tls1.2" (0x303)
    │                                   │            │    random{}:
0x00│                                 86│           .│      gmt_unix_time: 2249760024 (2041-04-16T21:20:24Z)
0x0c│18 9d 18                           │...         │
0x0c│         19 92 33 c2 21 ce 4f 97 30│   ..3.!.O.0│      random_bytes: raw bits
0x18│28 98 b3 fd 1e 15 f4 36 bb e9 14 f4│(......6....│
0x24│67 61 66 79 d5 3f 06               │gafy.?.     │
0x24│                     00            │       .    │    session_id_length: 0
    │                                   │            │    session_id: raw bits
0x24│                        c0 2f      │        ./  │    cipher_suit: "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" (0xc02f)
0x24│                              00   │          . │    compression_method: "null" (0x0)
0x24│                                 00│           .│    extensions_length: 20
0x30│14                                 │.           │
    │                                   │            │    extensions[0:2]:
    │                                   │            │      [0]{}: extension
0x30│   ff 01                           │ ..         │        type: "renegotiation_info" (65281)
0x30│         00 01                     │   ..       │        length: 1
0x30│               00                  │     .      │        data: raw bits
    │                                   │            │      [1]{}: extension
0x30│                  00 10            │      ..    │        type: "application_layer_protocol_negotiation" (16)
0x30│                        00 0b      │        ..  │        length: 11
0x30│                              00 09│          ..│        serer_names_length: 9
    │                                   │            │        protocols[0:1]:
    │                                   │            │          [0]{}: protocol
0x3c│08                                 │.           │            length: 8
0x3c│   68 74 74 70 2f 31 2e 31         │ http/1.1   │            name: "http/1.1"

# use ja3.jq to calculate ja3 TLS fingerprint
# https://github.com/wader/fq/blob/master/format/tls/testdata/ja3.jq
$ fq -L path/to/ja3 'include "ja3"; pcap_ja3' file.pcap
[
  {
    "client_ip": "192.168.1.193",
    "client_port": 64126,
    "ja3": "771,4866-4867-4865-49196-49200-159-52393-52392-52394-49195-49199-158-49188-49192-107-49187-49191-103-49162-49172-57-49161-49171-51-157-156-61-60-53-47-255,0-11-10-16-22-23-49-13-43-45-51-21,29-23-30-25-24,0-1-2",
    "ja3_digest": "bc29aa426fc99c0be1b9be941869f88a",
    "server_ip": "46.101.135.150",
    "server_port": 443
  }
]

toml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594
zip Properly decode EOCD record in zip64 files. Thanks @0-wiz-0 for report and spec interpretation. #586 #596
xml Fail faster to speed up probe. Could in some cases read the whole file before failing. Thanks @0-wiz-0 for report. #594

Changelog

0581ece Update docker-golang to 1.20.1 from 1.20.0
72870a5 Update docker-golang to 1.20.2 from 1.20.1
02e573a Update github-go-version to 1.20.1 from 1.20.0, 1.20.0, 1.20.0
c513088 Update github-go-version to 1.20.2 from 1.20.1
ce26372 Update github-golangci-lint to 1.51.1 from 1.51.0
75bfdda Update github-golangci-lint to 1.51.2 from 1.51.1
b1d9306 Update gomod-golang-x-crypto to 0.6.0 from 0.5.0
c03d3cc Update gomod-golang-x-crypto to 0.7.0 from 0.6.0
2430fba Update gomod-golang-x-net to 0.6.0 from 0.5.0
dd8ab79 Update gomod-golang-x-net to 0.7.0 from 0.6.0
80a0744 Update gomod-golang-x-net to 0.8.0 from 0.7.0
97643b9 Update gomod-golang/text to 0.7.0 from 0.6.0
e7168b9 Update gomod-golang/text to 0.8.0 from 0.7.0
36df57e Update make-golangci-lint to 1.51.1 from 1.51.0
70e08fa Update make-golangci-lint to 1.51.2 from 1.51.1
50d26ec colorjson: Handle encoding error value
5c8e115 colorjson: Refactor to option struct
8e0dde0 decode: Support multiple format args and some rename and refactor
a1bb630 doc,fq: Improve cli help and some cleanup
156aeec doc: Add FOSDEM 2023 talk
3e0ebaf doc: Run make doc
3cc8383 gojq: Update fq fork
dec433f help,markdown: Fix double line breaks when converting to text
c75a83c help: Show default option value as JSON
cc52a44 id3v2: Decode subframes for CTOC and add struct for headers
dc79a73 interp,json: Move error handling to colorjson
73db658 interp: Exit with error if -o name=@path fails to be read, also document
c8666ee ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg
b60acec matroska: Add decode_samples option
9aaf2dd matroska: Add unknown size test and add description to ebml header
a8d0bf4 matroska: Assume master with unknown size has ended if a valid parent is found
0d14d7b matroska: Handle unknown size for non-master types a bit better
c890a28 matroska: Update spec and make refs in descriptions look nicer
6c03245 pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6)
d4ea663 pcap: Add ipv4 fragments tcp test
f50bd6e readline: Update fq fork
9852f56 tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
56edb59 toml,xml: Fail fast on invalid content
5228fdd zip: Correctly look for and decode both zip32/64 EOCD record
bdd6718 zip: Correctly peek for zip64 EOCD

Contributors

path and 0-wiz-0

Assets 9

02 Feb 10:47

wader

v0.3.0

bafc8e3

v0.3.0

Bug fix release, no new features mostly due to holidays and busy with other things (some jq related!).

Also been preparing for a talk about fq this weekend at FOSDEM 2023.

Changes

TCP reassembly is now less strict about invalid TCP options. Turns out some options might end up wrong in packet captures due to hardware acceleration etc. For example it seems to be common that TCP segments end up larger than configured connection MSS. Now PCAP:s with those kinds of TCP segments should be reassembled correctly.
REPL now handles the del key properly. Before it could in some cases cause the output to be ignored.

Decoder changes

mp3 Add option for max unknown bits to handle more mis-probing. Default to 50%
mp4
- ftyp set minor description to date for "qt" files
- tkhd decode enabled, preview, etc flags
- udta Handle case wit box type is key and value rest of box
- sgpd,sbgp Change grouping type to a string as it seems to be what it is in practice.
tcp_segment Decode all standard options, MSS, Window scale, timestamp etc. Rename "maxseg" to "mss".

Changelog

8702e1d Update docker-golang to 1.19.5 from 1.19.4
a7f37d7 Update docker-golang to 1.20.0 from 1.19.5
826d9a5 Update github-go-version to 1.19.5 from 1.19.4, 1.19.4, 1.19.4
d338c8b Update github-go-version to 1.20.0 from 1.19.5, 1.19.5, 1.19.5
ad4919a Update github-golangci-lint to 1.51.0 from 1.50.1
e8ecbf9 Update gomod-golang/text to 0.6.0 from 0.5.0
f1057b9 Update make-golangci-lint to 1.51.0 from 1.50.1
ca27e42 doc: Add _parent for decode values and clenaup doc a bit
b04a650 flac_picture,mpeg: Fix trailing ")" typo in map sym and description
57144b2 github-action: Use quotes becase yaml (1.20 -> 1.2)
0aa6e3e gojq: Update rebased fq fork
7855b35 gomod: Update non-bump tracked mods and add bump config
6e17de3 goreleaser: Use name_template instead of deprecated archive replacements
8b49b42 interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format
2d82c05 mp3: Add max_unknown option to fail decode if too much unknown bits
f386a51 mp4: Decode qt minor verison as YYYY.MM description
3555dc6 mp4: Decode tkhd flags
c3e3b3e mp4: Decode udta metadata boxes without meta box
c49012d mp4: sgpd,sbgp: Change grouping_type to a string
6340365 mp4: udta: Handle box with value rest of box
55ef7a4 readline: Update fq fork to fix draw issue when using del key
1eb5e50 tcp: Ignore TCP option check for now as it seems unreliable in dumps
62e2cef tcp_segment: Decode standard options and rename maxseg to mss

Assets 9

23 Dec 13:35

wader

v0.2.0

63b5828

v0.2.0

This ended up being a release to cleanup old sins in the decoder internals and change some defaults how binary values work with JSON and string functions.

It also adds a new Time Zone Information Format decoder tzif (Thanks Takashi Oguma @bitbears-dev) and a new Apple BookmarkData decoder apple_bookmark decoder (Thanks David McDonald @dgmcdona). Also a new function from_ns_keyed_archiver was added to convert NSKeyedArchiver encoded objects into JSON.

A possible breaking change is that now all from/to prefix functions now has a from_/to_ prefix, ex: from_mp3 instead of frommp3. There are some few exceptions to this. Note that the functions named just be the format name, ex mp3 are still around.

In other fq related news jq-lsp got some fixed and additions and seems to work fine with neovim. It's also possible to use jq-lsp with vscode using vscode-jq.

Changes

All functions that had a from/to prefix now has the prefix from_/to_. This is to be easier to read and more consistent, there are still some exceptions like tovalue, torepr, tobytes etc but in general anything that does not deal with primitive types is now snake_case. #535
Change default bit_formats option value (how raw bits values are represented in JSON) from snippet to string. snippet meant truncated bits as base64. Now all bits are included as a UTF-8 string. The string will be binary safe (not lose any data) when used internally in fq but will lose data when represented in JSON as some bytes can't be encoded as UTF-8. #499

Don't auto convert to binary for string/regexp functions, turned out this is very confusing. Now you have to manually use tobytes etc to convert to binary value. #540

# This used to not work as test/1 would convert decode values to the source bytes
# (0x00 0x00 0x00 0x01) in this case. Now the jq value (symbolic in this case) will
# be used instead. You can do ".test | tobytes" to get old behavior.
#
# find all types with a "mdta." prefix
$ fq -o line_bytes=10 'grep_by(.type | test(`^mdta\.`))' file.mp4
     │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[0]{}: box
0x528│      00 00 00 1c            │  ....    │  size: 28
0x528│                  00 00 00 01│      ....│  type: "mdta.title" ("\x00\x00\x00\x01")
0x532│00 00 00 14 64 61 74 61 00 00│....data..│  boxes[0:1]:
0x53c│00 01 00 00 00 00 74 65 73 74│......test│
     │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[1]{}: box
0x546│00 00 00 25                  │...%      │  size: 37
0x546│            00 00 00 02      │    ....  │  type: "mdta.encoder" ("\x00\x00\x00\x02")
0x546│                        00 00│        ..│  boxes[0:1]:
0x550│00 1d 64 61 74 61 00 00 00 01│..data....│
0x55a│00 00 00 00 4c 61 76 66 35 39│....Lavf59│
0x564│2e 32 37 2e 31 30 30│        │.27.100│  │

Fix panic when cancel (ctrl-c etc) before interpreter is executing. Thanks @pldin601 for reporting. #495
Fix error using JQValue:s in assign/update paths, ex .[<JQValue here>] = 123 #509
Rename fields added for bit-ranges not used by a decoder from unknown# to gap#. "unknown" is probably a useful field name in some formats and "gap" describe better what it is. #500
Big decode API internals refactor to split scalars types into their own go types so they can store per type specific values. This also opens up for more ways to make fq both faster and more memory efficient. It also makes the decode API more type safe and makes it possible to experiment with decode DLS that uses chained methods etc. #523

Decoder changes

apple_bookmark New Apple BookmarkData decoder. Thanks David McDonald @dgmcdona. #493

bplist

Fix decoding of UID types
Adds a lost_and_found array with unused values
Fix an endian issue for unicode strings
Add NSKeyedArchiver to JSON helper function from_ns_keyed_archiver, see bplist docs for details on how to use it. Thanks David McDonald @dgmcdona. #493

# decode bplist, from_ns_keyed_archiver converts to JSON plist and then into object data as JSON, find app bookmarks keys and expand them as bookmark data and convert to represented JSON, and finally build path to applications
$ fq -r 'from_ns_keyed_archiver | (.. | .Bookmark? // empty) |= (apple_bookmark | torepr) | .. | .target_path? // empty | join("/")' recentapps.sfl2
System/Applications/Utilities/Terminal.app
Applications/Spotify.app
System/Applications/Calculator.app
System/Applications/Preview.app
Applications/Alacritty.app
Applications/DB Browser for SQLite.app
System/Applications/System Preferences.app
System/Library/CoreServices/Applications/Directory Utility.app
System/Applications/Utilities/Activity Monitor.app
Applications/Safari.app

tzif new Time Zone Information Format decoder. Thanks Takashi Oguma @bitbears-dev. #498

mp4

Map mdta metadata namespace and key names for ilst child boxes. #521

# create object with all ilst key/value pairs
$ fq 'grep_by(.type=="ilst").boxes | map({key: .type, value: .boxes[0].data}) | from_entries' file.mp4
{
  "mdta.encoder": "Lavf59.27.100",
  "mdta.title": "test"
}
# query specific value
$ fq -r 'grep_by(.type=="mdta.encoder").boxes[0].data | tovalue' file.mp4
Lavf59.27.100

Support sidx version 1. #506
Add description and symbolic values for traf sample flags, makes it easier to see and query for I-frames etc. #514

# which boxes has depends_on flags
$ fq 'grep_by(.sample_depends_on) | parent' fragmented.mp4

Support PNG codec mapping. #492
Decode pdin boxes. #524
Decode hnti boxes. #513

mp3_tags Add VBRI support and split into into mp3_frame_xing and mp3_frame_vbri decoders. #525

Changelog

7fa8b63 Add related file format projects to README
4fdb736 Update docker-golang to 1.19.4 from 1.19.3
519eff6 Update github-go-version to 1.19.4 from 1.19.3, 1.19.3, 1.19.3
2a91d29 Update gomod-golang/text to 0.5.0 from 0.4.0
cb15b37 added checks to prevent infinite looping and recursion
c244533 added some sfl2 test files to bplist package
7d13cf7 adds flag parsing to applebookmark
71b17d0 apple bookmarkdata decoder initial commit
8f39ef6 bplist: Harmonize ns_keyed_archive jq style a bit
cba72db bplist: added overload for from_ns_keyed_archiver jq func
129b4b7 bplist: doc: update docs to reflect changes to ns_keyed_archiver
9dab3c6 bplist: minor fix to from_ns_keyed_archiver
448c3ef bplist: update docs with from_ns_keyed_archiver reference, add error case to function
a9047c0 bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection
4a28e44 changes decoder package name from bookmark to apple_bookmark
d0b044c converts to snake_case and refactors decode helper
d199793 created stack type
e77f776 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
a85da29 decode: Make FieldFormat usage more consistent
9b81d4d decode: More type safe API and split scalar into multiple types
3ec0ba3 decode: add ns_keyed_archiver, restructure apple decoder into apple package
330d5f7 decode: apple_bookmark: simplifies flag decoding
93f2aa5 decode: change PosLoopDetector to use generics
7e98b53 decode: fix type on defer function call, test: add loop.fqtest
a873819 decode: fixes endian of unicode strings
f747873 decode: implements lost and found for unreferenced objects
b45f9fa decode: improve stack push/pop
a162e07 decode: minor change to method receiver name
3232f9c decode: moves PosLoopDetector into its own package
7c9504c decode: moves macho decoder to apple package
7083467 decode: remove dead code from ns_keyed_archiver
7ab4466 decode: remove unused field from decoder, unused parens from torepr
bdb8166 decode: removed unnecessary struct
98eab8c decode: rename parameter for consistency
04379df decode: revert decode.D back, place posLoopDetector in apple_bookmark
7fb674b decode: unexport methods
fa368bb decode: updates all.go with correct macho path
0287ffa decoding well but torepr needs work
42debe5 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
423bab9 dev,test: Use jqtest code from jqjq for jq tests
6fc84a8 doc,dev: Add more usage and dev tips
2fc16ae doc: Add some padding margin to formats table to make it less likely to cause git conflicts
62f377c doc: fixes snippet for recursive bookmark searching
22064f5 doc: remake
4aad2fd doc: remake
b872b1a doc: remake
1e1fc55 fixed one more snake_case letter
d0b76ca fixes broken test and removes long link from markdown body
5146f28 fixes broken test for all.fqtest
253033c fixes broken uid parsing in plist decoder
f535ad3 fixes spacing in jq files
64351e8 fixes tests and adds torepr test
c7d00b8 fixes unknown bit ranges
8f930aa forgot to add bookmark.jq in last commit
164e527 gojq: Update rebased fq fork
6c86945 gojq: Update rebased fq fork
578b84d interp,display: Add workaround for go 1.18 when escaping 0x7f
42d9f2c interp,help: Properly count line length when breaking on whole words
8d69f1f interp: Change default bits_format=string
6c229d7 interp: Don't auto convert to binary for string functions, is just confusing
568afff interp: Fix panic when trigger before any context has been pushed
e3ae144 interp: Rename to/from functions to to_/from_
ba88a68 interp: mimic jq: if expr arg is given read stdin even if tty
9bd65f9 migrates tests to per-sample files
f7d7a49 missed a letter on last commit - converting to snake_case
2f37cb5 mod: Up...

Contributors

pyldin601, dgmcdona, and bitbears-dev

Assets 9

Releases: wader/fq

v0.11.0

Changes

Format changes

Changelog

Contributors

v0.10.0

Changes

Format changes

Changelog

Contributors

v0.9.0

Changes

Decoder changes

Changelog

Contributors

v0.8.0

Changes

Decoder changes

Changelog

Contributors

v0.7.0

Changes

Decoder changes

Changelog

Contributors

v0.6.0

Changes

Decoder changes

Changelog

Contributors

v0.5.0

Changes

Decoder changes

Changelog

Contributors

v0.4.0

Changes

Decoder changes

Changelog

Contributors

v0.3.0

Changes

Decoder changes

Changelog

v0.2.0

Changes

Decoder changes

Changelog

Contributors