lib installation on Windows

[codeyourweb]

Hi,

I'm trying to use your library on Windows. I'm not familiar with C/C++ and linker/compilation problems but i've searched a lot before posting here. Maybe you could help me. First, i've installed mingw64 and also the latest version of visual studio. I've cloned the official yara repository and successfuly compiled libyara with visual studio. I saw that i've got the following folders:

• C:\YARA\windows\libyara (with a lot of *.obj files)
• C:\YARA\libyara (with a lot of *.c files)
• C:\YARA\libyara\include (with *.h files)
• C:\YARA\windows\vs2017\libyara\Release (with a 22MB libyara64.lib)

I didn't understood what i need to put in CGO_LDFLAGS and CGO_CFLAGS env vars or if i need additional compilations to successfully go get github.com/hillu/go-yara.

Actually it just return the following error:

# runtime/cgo
C:/mingw64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/8.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: cannot find -lyara

Any help would be appreciated :)

[hillu]

Sorry, I have no experience with using the Visual Studio buildsystem, so I can offer guesswork at best.

Assuming you want to use the VS-generated build using files from the paths you mention, I think that you'll have to

1. disable pkg-config based configuration since no meaningful yara.pc has been generated. This can be done with passing the build tag yara_no_pkg_config as described in README.md.
2. explicitly set CGO_CFLAGS so that the compiler looks in the correct directory for the header files: -IC:\YARA\libyra/include should be correct.
3. explicitly set CGO_LDFLAGS so that the linker looks in the correct directory for the library: -LC:\YARA\windows\vs2017\libyara\Release should be correct.
4. patch cgo so that libyara64 is used in your case (with pkg-config disabled). Try this:

   // #cgo yara_no_pkg_config                LDFLAGS:    -lyara64
   

   Perhaps passing the absolute filename instead of -lyara64 also works.

I hope this helps. If it does, I'd be very happy about a documentation pull request.

[codeyourweb]

definitly, i know why i'll never dev something in C. sometimes you take more times to successfully compile third-party libraries than coding your own program...

Anyway thanks for your help, it's seems to be better (even if i doesn't work for the moment). the linker seems to correctly parse my .lib file. However there are a lot of errors that i really don't how to deal with:

Link to error output

Have you got any idea on how to solve this?

[hillu]

Uh-oh. I never tried this before, but it seems like the Go toolchain can't link objects built with Visual C++ (see here). I think you are going to have to build libyara using MinGW.

[codeyourweb]

more complicated on windows than on linux but (with your help) i made progress. I installed MSYS2/mingw and managed to compile yara from bash. I gave up using pkg-config because it always returns an error, even from mingw. However running with tag "yara_no_pkg_config" is really better than before. It seems it's only missing one library. I've installed openssl and openssl-devel in mingw but it's not enough.

# go install --tags yara_static,yara_no_pkg_config
# github.com/hillu/go-yara/v4
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x1034): undefined reference to `MD5_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x114e): undefined reference to `MD5_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x119b): undefined reference to `MD5_Final'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3062): undefined reference to `EVP_sha1'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3092): undefined reference to `PKCS7_get0_signers'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x30bf): undefined reference to `OPENSSL_sk_num'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x30f6): undefined reference to `OPENSSL_sk_value'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3111): undefined reference to `X509_digest'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3165): undefined reference to `X509_get_issuer_name'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3176): undefined reference to `X509_NAME_oneline'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x31a2): undefined reference to `X509_get_subject_name'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x31b3): undefined reference to `X509_NAME_oneline'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x31e5): undefined reference to `X509_get_version'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3205): undefined reference to `X509_get_signature_nid'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x320c): undefined reference to `OBJ_nid2ln'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3251): undefined reference to `X509_get_serialNumber'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3263): undefined reference to `i2d_ASN1_INTEGER'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3277): undefined reference to `X509_get0_notBefore'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x329d): undefined reference to `X509_get0_notAfter'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x32cd): undefined reference to `OPENSSL_sk_num'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x32ee): undefined reference to `OPENSSL_sk_value'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3309): undefined reference to `OBJ_txt2nid'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3319): undefined reference to `X509at_get_attr_by_NID'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x3328): undefined reference to `X509at_get_attr'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x334e): undefined reference to `d2i_PKCS7'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x336c): undefined reference to `PKCS7_free'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x337e): undefined reference to `X509_ATTRIBUTE_get0_type'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x338d): undefined reference to `OPENSSL_sk_free'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x33b5): undefined reference to `i2d_ASN1_INTEGER'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x58f1): undefined reference to `OBJ_create'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x6d22): undefined reference to `PKCS7_free'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(pe.o):pe.c:(.text+0x6d55): undefined reference to `d2i_PKCS7'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xaa2): undefined reference to `MD5_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xc32): undefined reference to `MD5_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xd1e): undefined reference to `MD5_Final'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xecf): undefined reference to `MD5_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xede): undefined reference to `MD5_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xeee): undefined reference to `MD5_Final'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xfd5): undefined reference to `SHA256_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xfe4): undefined reference to `SHA256_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0xff4): undefined reference to `SHA256_Final'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x10df): undefined reference to `SHA1_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x10ee): undefined reference to `SHA1_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x10fe): undefined reference to `SHA1_Final'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x11fa): undefined reference to `SHA1_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x1398): undefined reference to `SHA1_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x148e): undefined reference to `SHA1_Final'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x164a): undefined reference to `SHA256_Init'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x17e8): undefined reference to `SHA256_Update'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: C:/msys64/usr/lib/libyara.a(hash.o):hash.c:(.text+0x18de): undefined reference to `SHA256_Final'
collect2.exe: error: ld returned 1 exit status

[hillu]

You need to pass flags to the linker so that it can find OpenSSL's libcrypto which contains the symbols mentioned in your error message: -L$PATH -lcrypto. PATH is probably the openssl-$VERSION directory that was created when unpacking the source distribution.

[hillu]

Can you post the error messages that pkg-config gave you?

[hillu]

Did the OpenSSL and YARA builds generate DLLs or only static libraries (lib*.a)?

[codeyourweb]

I'm currently at work, i'll check this when i'll be at home. Here's some elements that i remember:

• i don't remember the exact message of pkg-config but it throw an error because it can't found "m.pc".
• with mingw , yara was compiled as a *.a library (not *.lib like in VC++) , openssl il available in *.a and *.dll

Not sure to clearly understand your first comment. I need to pass this item to the linker like this, is it true?
CGO_LDFLAGS="-L${YARA_LIB_PATH} -lyara" "-L{$LCRYPTO_LIB_PATH -lcrypto}"
(with quotes and space separated or something like this?

[hillu]

Assuming you are setting CGO_LDFLAGS from within bash, use a single pair of double quotes, e.g.

export CGO_LDFLAGS="-L${YARA_LIB_PATH} -l yara -L${CRYPTO_LIB_PATH} -lcrypto"

[codeyourweb]

First of all, thank you for your help, because this time it works :) ... to be more precise, it NEARLY works, but maybe it's just my mistake and my lack of understanding of C linking process:

• compilation with CGO_LDFLAGS="-LC:/msys64/usr/lib -lyara -lcrypto" doesn't return any error :)

• running the project in your _examples folder return the same error as while i try to install without "no_pkg_config" tag

• About pkg-config, the exact error message is:

# pkg-config --cflags  -- yara
Package m was not found in the pkg-config search path.
Perhaps you should add the directory containing `m.pc'
to the PKG_CONFIG_PATH environment variable
Package 'm', required by 'yara', not found

pkg-config: exit status 1

maybe you'll know how to solve this error

• if we can't solve pkg-config issue, do you know to avoid pkg-config to be used? I've tried the following things but maybe i missed something:

1/ add a comment on first line of "simple-yara.go":
// #cgo yara_no_pkg_config LDFLAGS: -lyara
2/ run it with:
go run simple-yara.go --tags yara_no_pkg_config

[hillu]

Alright, we're getting there. It seems that the pkg-config error happens because of a bad commit in YARA:

commit a94f6deb1d2c60e12c6246c4562399c2e86f5a86
Author: Hilko Bengen <bengen@hilluzination.de>
Date:   Fri Nov 6 17:12:28 2020 +0100

    Add m (math) library to libs.private (pkg-config) if it is used
    
    This should get rid of "undefined reference to `log2'" errors when
    using pkg-config and using the static library.

You should be able to work around this by removing the "m" from the Requires.private line in the generated yara.pc file that is read by pkg-config.

I'll get this fixed properly (VirusTotal/yara#1408).

Please post the error messages you get from go run. I suspect that you didn't pass extra directories for the library search path (-L$path parameters).

[codeyourweb]

• i've compiled yara with your commit and removed the "m" in yara.pc. There isn't the previous error message anymore but an exit status 5-10 seconds after starting your example project:

C:\Users\myusername\go\src\github.com\hillu\go-yara\_examples\simple-yara> go run .
exit status 3221225781

• in order to run your example without pkg-config i've just edited the simple-yara.go like this:

package main
// #cgo !yara_no_pkg_config,!yara_static  pkg-config: yara
// #cgo !yara_no_pkg_config,yara_static   pkg-config: --static yara
// #cgo yara_no_pkg_config                LDFLAGS:    -lyara
import (
	"github.com/hillu/go-yara/v4"

	"bytes"
	"flag"
	"fmt"
	"log"
	"os"
	"path/filepath"
	"strconv"
	"sync"
)

func printMatches(item string, m []yara.MatchRule, err error) {
[...]

and then tried to run it like this (but it return the same error as if it's trying to run it with pkg-config and return the exact same error):
C:\Users\myusername\go\src\github.com\hillu\go-yara\_examples\simple-yara> go run . --tags yara_no_pkg_config

[hillu]

exit status 3221225781 could mean STATUS_DLL_NOT_FOUND (0xc0000135). This probably means that YARA or OpenSSL has been built as a DLL and not linked statically into the binary generated by go run. Try just running go build and inspect the import directory of the binary.

You can either pass -static to the linker or prevent the YARA and OpenSSL build scripts from generating DLLs in the first place.

[codeyourweb]

PS C:\Users\myusername\go\src\github.com\hillu\go-yara\_examples\simple-yara> go run .
2020/12/06 08:08:03 no rules specified
exit status 1

I've never been so happy to see an error from a golang code 😂 . As you see, it finally works! First of all, thanks for your help. Even if i worked a lot to deal with a mountain of problems installing this lib, i probably wouldn't have gotten there without your help.

My first mistake was to use msys2 terminal instead of mingw. Indeed, there was a problem with libcrypto and libssl (compiled as .a and .dll but source of conflicts. My second mistake was to directly clone VirusTotal yara repository to compile libyara. There are too much commit and it's the best way to deal with side effects. And last but not least, my third mistake was to compile libraries (like openssl) from source instead of leaving this task to the package manager (pacman on mingw).

I confirm that your library works on Windows, even without cross compile from a linux distribution. In few words it's necessary to:

• install msys2 (maybe i missed something with mingw64 / cygwin but i tried with these softwares and i got some errors)
• after msys2 fresh install , prefer using mingw terminal instead of msys one. (to avoid library install path conflict and missing libs)
• install dependencies pacman -S mingw-w64-x86_64-toolchain mingw-w64-x86_64-pkg-config base-devel openssl-devel
• configure environment variables (NOT in mingw/msys terminal but in your OS) (CGO_CFLAGS / CGO_LDFLAGS / PKG_CONFIG_PATH)
• compile libyara from the latest release (and not the latest commit)
• then install your librarie (could be done with windows cmd / powershell, no need of mingw terminal)

If you want, i can make a pull request with a specific documentation for Windows installation in a seperate txt/md file

[hillu]

Glad we finally got things worked out. 👍