Skip to content
This repository has been archived by the owner on Jun 7, 2023. It is now read-only.

Clarification - supported DB backends? #650

Open
frennkie opened this issue Feb 5, 2018 · 4 comments
Open

Clarification - supported DB backends? #650

frennkie opened this issue Feb 5, 2018 · 4 comments
Labels
bug (verified) Bug that has been verified
Milestone
2.0

Comments

@frennkie
Copy link
Contributor

frennkie commented Feb 5, 2018 •
edited

There are some issues that refer to database backends other than SQLite (#583 #573 #302).

The Viper documentation (https://github.com/viper-framework/viper/blob/master/docs/source/usage/concepts.rst) states that:

You can create as many projects as you want and you can easily switch from one to another. Each project will have its own local repositories of binary files, a SQLite database containing metadata

I tested this on PostgreSQL and also looked through the code. Having more than one database currently only works automatically with SQLite. A workaround is to manually edit the viper.conf, replace the connection string and restart Viper. viper-update currently also only takes care of SQLite files.

So either we explicitly state that users MUST use SQLite or we add the missing things to the roadmap. What do you think @botherder @Rafiot @deralexxx ?
@frennkie frennkie mentioned this issue Feb 18, 2018
Merged
@frennkie
Copy link
Contributor Author

frennkie commented Feb 18, 2018 •
edited

Here is a log of a session using a PostgreSQL database and trying to use different projects:

$: tree /tmp/testing
/tmp/testing
├── viper_case1
│   ├── file5
│   ├── file6
│   └── file7
├── viper_case2
│   ├── file8
│   └── file9
└── viper_default
    ├── file1
    ├── file2
    ├── file3
    └── file4

3 directories, 9 files

$: ./viper-cli
/home/vipert/work/viper/venv/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8;
in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
         _
        (_)
   _   _ _ ____  _____  ____
  | | | | |  _ \| ___ |/ ___)
   \ V /| | |_| | ____| |
    \_/ |_|  __/|_____)_| v1.3-dev
          |_|

You have 0 files in your default repository
viper > about
+----------------+-------------------------------------------------+
| About          |                                                 |
+----------------+-------------------------------------------------+
| Viper Version  | 1.3-dev                                         |
| Python Version | 3.5.2                                           |
| Homepage       | https://viper.li                                |
| Issue Tracker  | https://github.com/viper-framework/viper/issues |
+----------------+-------------------------------------------------+
+--------------------+--------------------------------------------------+
| Configuration      |                                                  |
+--------------------+--------------------------------------------------+
| Configuration File | /home/vipert/work/viper/viper/viper.conf         |
| Active Project     | default                                          |
| Storage Path       | /home/vipert/.viper                              |
| Database Path      | postgresql://viper:changeme@localhost:5432/viper |
+--------------------+--------------------------------------------------+

viper > store -f /tmp/testing/viper_default
[+] Stored file "file4" to /home/vipert/.viper/binaries/d/6/d/1/d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e
[*] Session opened on /home/vipert/.viper/binaries/d/6/d/1/d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e
[*] Running command "yara scan -t"
[*] Scanning file4 (d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e)
[*] Running command "triage"
[+] Stored file "file3" to /home/vipert/.viper/binaries/7/4/6/5/7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36
[*] Session opened on /home/vipert/.viper/binaries/7/4/6/5/7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36
[*] Running command "yara scan -t"                                                                                                                           
[*] Scanning file3 (7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36)
[*] Running command "triage"
[+] Stored file "file1" to /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
[*] Session opened on /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
[*] Running command "yara scan -t"
[*] Scanning file1 (8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801)
[*] Running command "triage"
viper > projects -l
[*] The projects directory does not exist yet
viper > projects -s case1
[*] Switched to project case1
case1 viper > store -f /tmp/testing/viper_case1
[+] Stored file "file7" to /home/vipert/.viper/projects/case1/binaries/b/9/1/4/b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae
[*] Session opened on /home/vipert/.viper/projects/case1/binaries/b/9/1/4/b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae
[*] Running command "yara scan -t"
[*] Scanning file7 (b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae)
[*] Running command "triage"
[+] Stored file "file6" to /home/vipert/.viper/projects/case1/binaries/4/c/f/f/4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86
[*] Session opened on /home/vipert/.viper/projects/case1/binaries/4/c/f/f/4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86
[*] Running command "yara scan -t"
[*] Scanning file6 (4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86)
[*] Running command "triage"
[+] Stored file "file5" to /home/vipert/.viper/projects/case1/binaries/4/5/4/0/454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae
[*] Session opened on /home/vipert/.viper/projects/case1/binaries/4/5/4/0/454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae
[*] Running command "yara scan -t"
[*] Scanning file5 (454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae)
[*] Running command "triage"
case1 viper > projects -s case2
[*] Switched to project case2
case2 viper > store -f /tmp/testing/viper_case2
[+] Stored file "file9" to /home/vipert/.viper/projects/case2/binaries/8/3/0/d/830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da
[*] Session opened on /home/vipert/.viper/projects/case2/binaries/8/3/0/d/830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da
[*] Running command "yara scan -t"
[*] Scanning file9 (830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da)
[*] Running command "triage"
[+] Stored file "file8" to /home/vipert/.viper/projects/case2/binaries/5/8/6/6/5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7
[*] Session opened on /home/vipert/.viper/projects/case2/binaries/5/8/6/6/5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7
[*] Running command "yara scan -t"
[*] Scanning file8 (5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7)
[*] Running command "triage"
case2 viper >
case2 viper > projects -l
[*] Projects Available:
+--------------+--------------------------+---------+
| Project Name | Creation Time            | Current |
+--------------+--------------------------+---------+
| case1        | Sun Feb 18 14:52:22 2018 |         |
| case2        | Sun Feb 18 14:52:37 2018 | Yes     |
+--------------+--------------------------+---------+
case2 viper > projects -s default
[*] Switched to project default
default viper > projects -l
[*] Projects Available:
+--------------+--------------------------+---------+
| Project Name | Creation Time            | Current |
+--------------+--------------------------+---------+
| case1        | Sun Feb 18 14:52:22 2018 |         |
| case2        | Sun Feb 18 14:52:37 2018 |         |
+--------------+--------------------------+---------+
default viper > find all
+---+-------+--------------------------+----------------------------------+------+
| # | Name  | Mime                     | MD5                              | Tags |
+---+-------+--------------------------+----------------------------------+------+
| 1 | file4 | application/octet-stream | 2a70104edc73f0ea5e450dab92fc5a11 |      |
| 2 | file3 | application/octet-stream | 13e277d58cfb358da9a6e25634910d7d |      |
| 3 | file1 | application/octet-stream | addcac07a641ea4f657b39996768e4c9 |      |
| 4 | file7 | application/octet-stream | 2631254381f21571b1728b8eaa5fd040 |      |
| 5 | file6 | application/octet-stream | b01a627f828a400e3eefa69373281e23 |      |
| 6 | file5 | application/octet-stream | 88f3ef847927c0c3a16aa6a509eca111 |      |
| 7 | file9 | application/octet-stream | 0d1bd8cdcb8e41370cfc24ea6f255de7 |      |
| 8 | file8 | application/octet-stream | 50bbc877fb1a0a7c2104935e763ae9eb |      |
+---+-------+--------------------------+----------------------------------+------+
default viper > projects -s case1
[*] Switched to project case1
case1 viper > find all
+---+-------+--------------------------+----------------------------------+------+
| # | Name  | Mime                     | MD5                              | Tags |
+---+-------+--------------------------+----------------------------------+------+
| 1 | file4 | application/octet-stream | 2a70104edc73f0ea5e450dab92fc5a11 |      |
| 2 | file3 | application/octet-stream | 13e277d58cfb358da9a6e25634910d7d |      |
| 3 | file1 | application/octet-stream | addcac07a641ea4f657b39996768e4c9 |      |
| 4 | file7 | application/octet-stream | 2631254381f21571b1728b8eaa5fd040 |      |
| 5 | file6 | application/octet-stream | b01a627f828a400e3eefa69373281e23 |      |
| 6 | file5 | application/octet-stream | 88f3ef847927c0c3a16aa6a509eca111 |      |
| 7 | file9 | application/octet-stream | 0d1bd8cdcb8e41370cfc24ea6f255de7 |      |
| 8 | file8 | application/octet-stream | 50bbc877fb1a0a7c2104935e763ae9eb |      |
+---+-------+--------------------------+----------------------------------+------+
case1 viper > open -l 3
[!] You have to open a session on a path or on a misp event.
case1 viper > open -l 9
case1 viper > info
case1 viper > projects -s case2
[*] Switched to project case2
case2 viper > open -l 9
case2 viper > info
case2 viper > projects -s default
[*] Switched to project default
default viper > open -l 9
default viper > info
default viper > about
+----------------+-------------------------------------------------+
| About          |                                                 |
+----------------+-------------------------------------------------+
| Viper Version  | 1.3-dev                                         |
| Python Version | 3.5.2                                           |
| Homepage       | https://viper.li                                |
| Issue Tracker  | https://github.com/viper-framework/viper/issues |
+----------------+-------------------------------------------------+
+--------------------+--------------------------------------------------+
| Configuration      |                                                  |
+--------------------+--------------------------------------------------+
+--------------------+--------------------------------------------------+
| Configuration File | /home/vipert/work/viper/viper/viper.conf         |
| Active Project     | default                                          |
| Storage Path       | /home/vipert/.viper                              |
| Database Path      | postgresql://viper:changeme@localhost:5432/viper |
+--------------------+--------------------------------------------------+
default viper > open -l 3
[*] Session opened on /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
default viper file1 > info
+----------+----------------------------------------------------------------------------------------------------------------------------------+
| Key      | Value                                                                                                                            |
+----------+----------------------------------------------------------------------------------------------------------------------------------+
| Name     | file1                                                                                                                            |
| Tags     |                                                                                                                                  |
| Path     | /home/vipert/.viper/binaries/8/e/c/0/8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801                            |
| Size     | 2097152                                                                                                                          |
| Type     | data                                                                                                                             |
| Mime     | application/octet-stream                                                                                                         |
| MD5      | addcac07a641ea4f657b39996768e4c9                                                                                                 |
| SHA1     | fe367a4d8e166b7072ab579fc46ed66a4c643c49                                                                                         |
| SHA256   | 8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801                                                                 |
| SHA512   | e5944e845b826a69985c2305de8740969951b34262694a2bd62416c5ff55d7a85eb385dc9242f4b58d033f7cb38ae352b3fe3c1f28193aa2b901d98d8a4beb0c |
| SSdeep   | 49152:lhUkYoVnJFOm1x0PSyCFCbGPIT8kTPj7KW+Rw4jNlQQuS:Z/nfNxOhbGPIYqr8lQQuS                                                        |
| CRC32    | 7AE90063                                                                                                                         |
| Parent   |                                                                                                                                  |
| Children |                                                                                                                                  |
+----------+----------------------------------------------------------------------------------------------------------------------------------+
default viper file1 >

$: tree /home/vipert/.viper
/home/vipert/.viper
├── binaries
│   ├── 7
│   │   └── 4
│   │       └── 6
│   │           └── 5
│   │               └── 7465b8337e40016ec17c5e3be6377ec4358cafb35db81d6b24dbd2ff29447c36
│   ├── 8
│   │   └── e
│   │       └── c
│   │           └── 0
│   │               └── 8ec0e7ae18a08cb4bcbb659e6efcecae30f6be5e16e6cfb425d938d2b1701801
│   └── d
│       └── 6
│           └── d
│               └── 1
│                   └── d6d17244b216e2490b565c70683624accf42f72a30873bd5ed8a457e16eebb0e
├── history
├── projects
│   ├── case1
│   │   └── binaries
│   │       ├── 4
│   │       │   ├── 5
│   │       │   │   └── 4
│   │       │   │       └── 0
│   │       │   │           └── 454012b57d582deb8d6b048cec77cd0f331c0e8c5742638caeab899345f275ae
│   │       │   └── c
│   │       │       └── f
│   │       │           └── f
│   │       │               └── 4cfffcc1f1d82d642c9325ecab7168055cf0dde16e7a872a27a256aed8976d86
│   │       └── b
│   │           └── 9
│   │               └── 1
│   │                   └── 4
│   │                       └── b914d94373e431658832a84a46712d77d7acaff93c6eacbc1c996886c0f25dae
│   └── case2
│       └── binaries
│           ├── 5
│           │   └── 8
│           │       └── 6
│           │           └── 6
│           │               └── 5866438985c4f41a1fe70bd08cf018fcaa6dd6ee075efb8269c813abe99eded7
│           └── 8
│               └── 3
│                   └── 0
│                       └── d
│                           └── 830d0a80fb1df2d8f9a78e6d223c3d1e0b7add927cfcaf8897e650bc216af0da
├── scraper
└── viper.log

38 directories, 10 files

So this is definitely broken.

@jaegeral
Copy link
Contributor

Agree, we should than change the documentation to sqlite is the only one at the moment really working

@botherder botherder added this to the 2.0 milestone Mar 22, 2018
@botherder botherder added the bug (verified) Bug that has been verified label Mar 22, 2018
@botherder
Copy link
Member

I'm not sure I understand the problem. If by changing the connection configuration to specify another type of DBMS everything works fine, it is intended behavior. No?

@frennkie
Copy link
Contributor Author

Hm.. I think my main concern is that the behavior when using SQLite is documented and easily to understand... you run ./viper-cli and can switch between projects. Each project has it's own database and the binaries are in a separate folder per project.

But if you use a different DBMS you have to exit viper, change the .conf file and then run viper-cli -p caseA to make sure that the selected database matches the binaries found in the project dir.

If this is the intended behavior then this should at least be documented (currently there is not mention of changing connection in the concepts.rst doc).

@botherder botherder mentioned this issue Dec 19, 2018
Closed
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug (verified) Bug that has been verified
Projects
None yet
Milestone
2.0
Development

No branches or pull requests
3 participants
@jaegeral @botherder @frennkie