Dnn2

[pfeatherstone]

So still working on dnn2. Here is my proposal to really bring down compile time.

1. Layer details like relu_, con_ etc will not be templated. All parameters will be defined at runtime. So constructors will take all the parameters
2. Similarly to add_layer in normal dnn module, I will have a buffered_layer which will have all the common state. Rather than it be templated on the layer details (like add_layer does), it will erase the layer using type erasure. I will have a small buffer optimization designed so most layers are stored on the stack. Otherwise if they overrun the SBO, they will be stored on the heap. This means that buffered_layer is not a template anymore.
3. The model definition will be a list of lambdas which constructs the entire model. E.g.

auto model = module(input(0,0,0), 
                    con(/*outc=*/ 32, /*nr=*/ 3, /*nc=*/ 3, /*stride_y=*/ 1, /*stride_x=*/  1, /*padding_y=*/ 1, /*padding_x=*/ 1),
                    bn(32),
                    leaky_relu(),
                    ...,
                    loss(/*args*/));

Once evaluated, it will just return something that wraps std::vector<buffered_layer>. And you will construct it like so:

auto net = model(/*args*/);

4. I propose to drop affine, and simply define .eval() and .train() like in torch. This will only affect layers like bn_ and dropout_.

I think this will substantially reduce compile times without having to do too much of a re-write (I don't need a torch like tensor class with autograd or anything like that. Most of the layer details are copy-pasted).
The API will look very similar. Indeed,

using model = loss<...,leaky_relu<bn<con,input>>>>;

is not too different to:

auto model = module(input(/*args*/), con(/*args*/), bn(/*args*/), leaky_relu, loss);

[pfeatherstone]

Notice, layers will be defined left to right, not right to left! This seems more sensible

[pfeatherstone]

A repeated layer will be used like so:

auto model = module(input(), repeat(con(), 4), loss());

[pfeatherstone]

For reusable blocks, i expect to be able to do this:

auto conblock = [](size_t nfilters, size_t nr = 3, size_t nc = 3, ...) {return module(con(nfilters,nr,nc), bn(nfilters), relu);};
auto model = module(input(), conblock(32) repeat(conblock(32), 4), loss());

Something like that. It's definitely doable.

[pfeatherstone]

The main reason for doing this is that compilers are insanely good at compiling lambdas, it's ridiculous. So using them would be a win. Then having the lambdas collapse to something that wraps std::vector when invoked means the symbol names for the models becomes super short and always the same, no matter how complicated your model is.

If we're clever we could get it to reduce down to std::array. Then if the SBO size is set to something large-ish like 256, then pretty much everything would be on the stack. Maybe the SBO could be configurable so that on systems with small stack sizes, the heap is used more. In a first instance i will stick to using std::vector

[pfeatherstone]

Example usage of tags for resnet blocks:

auto convblock = module(tag(0), con(), bn(), relu, add(0));

something like that

[pfeatherstone]

Automatic serialization is trivially done, so no worries there.

[davisking]

Eh, IDK. I think it's probably less work. I guess it depends on the details. And auto-grad is kinda trivial. It's just backprop. The tedious part is implementing all the ops with their gradients but that's already done.

But yeah, maybe it's more work. You will end up making a new trainer class and all the other layer stuff though. And there are a bunch of odd bits like tag, repeat, skip, and all of that that are only needed in the current setup. With the torch way it's all really simple. There is just some class that represents the output of an operation and it has shared pointers pointing to the input operations. That class is basically a tensor with this extra book keeping.

Like how are you going to deal with ops that want to take inputs from more than the previous layer? That's just automatically trivial in the torch way, but complicated in the current dlib setup. And in your new setup it's not super obvious to me how that would work out, since the layers don't get generic access to the subnetwork right?

[pfeatherstone]

That's all handled when iterating through the layers. I've got that working currently but without the type erasure stuff. I'll start committing stuff soon so you can see. I think this is a good stepping stone. It solves the compilation issues and the API looks and feels largely the same. I agree the torch way is better but I wouldn't know where to start to implement it cleanly. Maybe you're better suited. But I'm guessing you don't have the time.

[pfeatherstone]

Also if there is going to be a torch like API I think supporting multi-dimensional arrays of arbitrary rank would be really great. I'm guessing that's not trivial. It would be also very cool if at runtime you could switch between cpu and gpu device. This is all quite a bit of work I imagine. With this proposal, I'm not pretending to tackle any of that, just make dlib dnn models practical compile time wise

[davisking]

Ok, sounds good.

Yeah, I've got other things I need to devote my time to right now :| I'll go back to writing more open source more often eventually. It's going to be a while though.

[pfeatherstone]

To be honest, I wish I could use libtorch. But you can only use the pre-built libraries. I'm not a fan of that. To build it from source you need A LOT of RAM and i've only every successfully built it from source once. Except for that one time, it always crashed. I love to be able to integrate libraries in cmake either via FetchContent or add_subdirectory and build from source locally. Dlib provides that so this is why I would like to make it a bit more research-friendly.

[arrufat]

Hi @pfeatherstone I just saw this episode of C++ Weekly released about type erasure.

I was thinking if the idea of having some kind of layer_view like he suggests in the video (animal_view) would also make sense for this.

Maybe it's already what you're doing, I didn't have time to go through your PR.

[pfeatherstone]

Yeah we could either provide an interface which can return a view of a layer or a simple lvalue reference. The advantage of a view is that it's impossible to copy the underlying layer that's type erased. Don't know. Could do both. But yes, the current PR supports storage_view which would allow you to do this

[pfeatherstone]

Been a while since i gave an update. I stopped working on this weeks ago. I've really got into transformers lately (they are awesome!). I don't think dlib can support transformers without substantial changes to the core. So i lost interest in this since i don't think i'll ever use dlib for neural nets again unless it's a super simple classifier.

[pfeatherstone]

I think i agree with @davisking now that there's not much point working on a "reduced-compile-time" API. The next version should be a bit of a redesign with a focus on tensor support, autograd and an "eager-mode" functional api a bit like JAX or pytorch. But I think the torch team are working on both torch.compile() and torch.export() which will hopefully make it easier to deploy models in C++ without too much overhead. We'll see

[davisking]

Yeah, like honestly I use pytorch at work and it's nice for this stuff. Just use that :)

[Compaile]

Hi,
we are not very happy with torch atm.

Are there any updates or is there any pilot code we could use to try finish the dnn2 rework.

[davisking]

Na no work on this yet.

[pfeatherstone]

I'm gonna be honest I'll probably not work on this. I don't see the value-add anymore given the number of edge inference libraries out there now (I was only ever going to use dlib for inference, not training) Furthermore, a good inference library needs to support bf16 and int8, which dlib doesn't. Nowadays you can even compile your models down to machine code so you don't even need a library to run models. So yeah, I'll probably won't work on this.

[davisking]

Yeah that's my feeling on it as well.