Is there a lightgrep release?

[simsong]

bulk_extractor for windows previously downloaded this release of lightgrep and built it:
https://github.com/LightboxTech/liblightgrep/archive/v1.3.0.tar.gz

Unfortunately, the release is now gone. There are no releases in https://github.com/strozfriedberg/lightgrep. Is there going to be a lightgrep release?

[sf-jonstewart]

Hey, Simson! For the moment, you can find the release here: https://github.com/strozfriedberg/liblightgrep/releases.

We have somewhat recently merged the "liblightgrep" and "lightgrep" repositories and will be working off the "lightgrep" one now. We're in the midst of some lightgrep work at the moment. Some [small] breaking API changes have been made and some ASCII/Unicode/casefolding issues addressed. Hopefully there will be some performance improvements coming soon. We'll make a PR for bulk_extractor when it's ready and let you know about new Windows binaries. Bulk_extractor is in our build pipeline, so we keep on top of any necessary changes.

Is https://github.com/simsong/bulk_extractor still the master repository for PRs?

cheers,

Jon

[simsong]

Hi Jon! Yes, https://github.com/simsong/bulk_extractor is still the master bulk_extractor PR. I am putting out the 1.6.0 release. It's pretty much done. There are some new scanners. Let me know when Lightgrep is stable again and we'll incorporate it in the 1.6.x release. I now have VM configuration for AWS Linux, Ubuntu, Centos, and Fedora 31. (Can you believe it's up to 31?) GCC version 9 threw a whole bunch of warnings which I corrected. I found a crashing but if scan_wordlist that nobody had ever reported before. All in all, though, it's pretty stable. The big thing that's needed is a new UI that's not based on Java; I think that either JavaScript or something that runs inside a Autopsy would be the way to go. Do you know of anyone at SF who would be interested in doing it?

[sf-jonstewart]

This v1.3 build should be plenty stable: https://github.com/strozfriedberg/liblightgrep/archive/v1.3.0.tar.gz. It dates from 2014 and my guess is it's the exact same bits. (I was recently musing how I should follow your example with simson.net and archive everything I work on religiously, because life is long and it's easy to misplace things after a few years.)

I'd encourage you to try out v1.4, though: https://github.com/strozfriedberg/liblightgrep/archive/v1.4.tar.gz. It's from the fall of 2014 and has some significant performance improvements over 1.3. If it doesn't work right out of the box, let us know and we can fix things up.

I agree that a separate Java UI for bulk_extractor is a burden. Autopsy support solves the problem nicely, but only for a particular user segment. It's still worth doing, probably, and might be possible as an Autopsy plugin. For a web UI, Python's Flask project can let you stand up a simple server. If you want to be front-end only, with no server at all, then Vue.js might be helpful.

We don't presently do any development-for-hire work, and my team isn't skilled at front-end web development. So, doubtful we'd be in much of a position to help.

[simsong]

bulk_extractor has a "server mode" that supports the forensic path translation for the GUI; it would be easy to turn that into an HTTP server and use it for the GUI. No need for flask. I'll check out Vue.js. I'll also try v1.4.

As for long-term archiving — simson.net doesn't have everything. There's also stuff on github and elsewhere. I do try to archive stuff at simson.net/ref//. It will be searchable with a MediaWiki plugin that should be operational by the end of the year.

[simsong]

It would be great if, in the future, the filenames were things like lightgrep_1.3.0.tar.gz and not v1.3.0.tar.gz.

[simsong]

Well, v1.4 doesn't install on Fedora 31, even when icu is installed:

...
checking if bison is the parser generator... yes
bison version is... bison (GNU Bison) 3.4.1
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for ICU... no
configure: error: Package requirements (icu-uc) were not met:

Package 'icu-uc', required by 'virtual:world', not found

Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.

Alternatively, you may set the environment variables ICU_CFLAGS
and ICU_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
[user@localhost liblightgrep-1.4]$ icu
icupkg
[user@localhost liblightgrep-1.4]$ icupkg
usage: icupkg [-h|-?|--help ] [-tl|-tb|-te] [-c] [-C comment]
	[-a list] [-r list] [-x list] [-l [-o outputListFileName]]
	[-s path] [-d path] [-w] [-m mode]
	[--auto_toc_prefix] [--auto_toc_prefix_with_type] [--toc_prefix]
	infilename [outfilename]
[user@localhost liblightgrep-1.4]$ icupkg -l
usage: icupkg [-h|-?|--help ] [-tl|-tb|-te] [-c] [-C comment]
	[-a list] [-r list] [-x list] [-l [-o outputListFileName]]
	[-s path] [-d path] [-w] [-m mode]
	[--auto_toc_prefix] [--auto_toc_prefix_with_type] [--toc_prefix]
	infilename [outfilename]
[user@localhost liblightgrep-1.4]$

[sf-jonstewart]

It's old enough that pkg-config expectations for ICU might have changed in the intervening year. We will spin up a Fedora31 system and look at it first thing in the morning. On Nov 11, 2019, at 8:46 PM, Simson L. Garfinkel <notifications@github.com<mailto:notifications@github.com>> wrote: Well, v1.4 doesn't install on Fedora 31, even when icu is installed: ... checking if bison is the parser generator... yes bison version is... bison (GNU Bison) 3.4.1 checking for pkg-config... /usr/bin/pkg-config checking pkg-config is at least version 0.9.0... yes checking for ICU... no configure: error: Package requirements (icu-uc) were not met: Package 'icu-uc', required by 'virtual:world', not found Consider adjusting the PKG_CONFIG_PATH environment variable if you installed software in a non-standard prefix. Alternatively, you may set the environment variables ICU_CFLAGS and ICU_LIBS to avoid the need to call pkg-config. See the pkg-config man page for more details. [user@localhost liblightgrep-1.4]$ icu icupkg [user@localhost liblightgrep-1.4]$ icupkg usage: icupkg [-h|-?|--help ] [-tl|-tb|-te] [-c] [-C comment] [-a list] [-r list] [-x list] [-l [-o outputListFileName]] [-s path] [-d path] [-w] [-m mode] [--auto_toc_prefix] [--auto_toc_prefix_with_type] [--toc_prefix] infilename [outfilename] [user@localhost liblightgrep-1.4]$ icupkg -l usage: icupkg [-h|-?|--help ] [-tl|-tb|-te] [-c] [-C comment] [-a list] [-r list] [-x list] [-l [-o outputListFileName]] [-s path] [-d path] [-w] [-m mode] [--auto_toc_prefix] [--auto_toc_prefix_with_type] [--toc_prefix] infilename [outfilename] [user@localhost liblightgrep-1.4]$ — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#5?email_source=notifications&email_token=ADUPJC6AHVTZE7F6ILSZ7YLQTIDGVA5CNFSM4JL2OQX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDYXS2I#issuecomment-552696169>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADUPJC7AGQSWSDQK4JAA4YLQTIDGVANCNFSM4JL2OQXQ>.

[simsong]

With GCC 9.x, Fedora 31 is my dream release.
("A dream to some... " https://www.youtube.com/watch?v=wuTviZDhXEE)

[uckelman-sf]

I had no problems building liblightgrep 1.4 on Fedora 31.

Simson, do you have libicu-devel installed? (I suspect that you're missing libicu-devel and also boost, because libicu-devel is one of its dependencies.)