embedding cpp! in other macros

[fneddy]

The readme says that it is not possible to embed the cpp! macro in other macros. What is needed to make this possible?

[mystor]

We'd need more APIs and ways to interact with the compiler. Currently we use a build script to collect all header includes and individual cpp! calls into C++ code which is built and linked with the rust code, and then during the rust compilation those cpp! calls are converted into actual FFI calls into that pre-built C++ code.

We'd need to be able to generate that C++ code during macro expansion while the rustc compiler is running, and hook into the compiler before the link step in order to compile and link the C++ code into a final library.

Right now we also take advantage of the fact that we've built all of the C++ code before we start compiling rust code by making the rust procedural macro insert assertions that object size and alignment matches across the FFI boundary, and that would be much trickier to do if we can't do the pre-build work we're doing right now.

It might be possible if we don't care about invoking the C++ compiler once for each cpp! closure invocation. (all header includes are parsed in the build script but the actual code being parsed by the invocation), but I'm not sure.

All in all, it's a lot of work with tradeoffs, but it might be doable.

[frankier]

I've been using these crates for wrapping some C++ libraries I need to use from Rust. This library has been the easiest, lowest anxiety way to achieve it, so thanks very much for your work so far.

Just to provide a use case for this, I end up with a certain pattern a lot when Rust ends up with ownership of C++ object:

pub struct CPlusPlusBox {
    payload: *mut c_void,
}

impl CPlusPlusBox {
    pub fn new(...) -> CPlusPlusBox {
        unsafe {
            result = cpp!([...] -> *mut c_void as "MyCPlusPlusType*" {
                return new MyCPlusPlusType(...);
            });
        }
        result
    }
}

impl Drop for CPlusPlusBox {
    fn drop(&mut self) {
        let payload = self.payload;
        unsafe {
            cpp!([payload as "MyCPlusPlusType*"] {
                delete payload;
            });
        }
    }
}

I was hoping I could factor out some of this boilerplate using a macro (or perhaps a custom derive might make more sense in this particular case). In general, allowing for macros to generate cpp! blocks could make writing wrappers less boilerplate heavy giving as a sort of middle way between the current situation (more manual work, lots of control, no problem with using all C++ features) and bindgen (more automation, less control, confusing to deal with templates).

I'm a still more or less Rust neophyte (for example, I haven't yet actually even written a macro myself - this is the first time I've felt the need) and might be out of my depth here, and perhaps I don't fully understand the issues completely, but I think perhaps I might have a potential slightly slow/hacky way this could maybe be implemented reasonably easily:

1. First somehow replace/monkeypatch the cpp![]{} macro with a no-op, for example by playing with the paths the Rust compiler looks in for sources.
2. Do an initial macro expansion of each Rust file before looking for cpp! blocks using "rustc --pretty expanded".
3. Put the actual cpp![]{} macro back, and proceed as normal.

Might something like this work? One immediate problem that jumps out here is error messages and keeping track of line numbers might become challenging. In particular, even if it's possible to retrieve the pre-expansion line numbers, what about when there's an error in some C++ generated by a macro. How can it be attributed to the correct macro expansion? Taking a preliminary look, it's clear you've worked quite hard to get proper line numbering so I guess this is definitely something which needs to be addressed properly.

[vadimcn]

Asserts can be handled by generating a companion global struct for each c++ closure, which would store size/alignment of all of its arguments.

The tricky bit seems to be knowing when to invoke the c++ compiler. Proc-macro crate interface currently has no provisions for rustc notifying the macro crate that it is done with macro expansion. This could be worked around, though, by asking users to place generate_cpp!() macro as last thing in crate's main file (which would expand to a custom derive from cpp_macros). I think it's pretty safe to assume that such a macro will be expanded last.

In the future, when proc_macro_attribute is stable, this could become simply a crate-level attribute.

[mystor]

Unfortunately, I don't think that you can make custom crate-level attributes with proc_macro_attribute, as you can't import the macro to use it before the attribute needs to be resolved. I think there used to be an ICE which was caused by trying that, which was fixed by simply disabling it. I might be wrong / things may have changed since then though.

I'm not comfortable depending on the order of macro expansion right now, especially because there's nothing technically stopping rustc from performing parallel macro expansion, which may become a thing which we do in the future for performance reasons, especially if people start doing a lot of complex work in procedural macros.

In general I think the easiest way to handle code which is generated by macros would be to try to discover all of the files ahead of time & build them, like today, but have a fallback where each individual macro which wasn't discovered is built separately if you're using macros, and we link in all of the many small crates. I think it might work OK, but we wouldn't be able to support expanding includes etc. from of macros. It might end up being a thing which is only enabled if you enable a feature.

We already generate a global struct for each c++ closure which stores size/alignment of all arguments - we just check it at compile time instead of at run time to try to avoid the performance overhead of actually performing the asserts at runtime when they are trivially true.

[ogoffart]

#35 will help in some cases (although it won't help for the CPlusPlusBox use case, but that's covered by cpp_class! )

[vadimcn]

What if build script used rustc itself to parse the crate? (with a different implementation of the cpp! macro to collect c++ code).

Additionally, I am thinking that rust-cpp might be able to get rid of as "<c++ type>" annotations if we could find a way to embed type metadata into rlib generated in this step.

Perhaps something like this:

pub trait CPPTypeInfo {
    const CPP_TYPE: &'static str;
    const SIZE: usize;
}

impl CPPTypeInfo for i32 {
    const CPP_TYPE: &'static str = "int32_t";
    const SIZE: usize = size_of::<i32>();
}
impl CPPTypeInfo for f32 {
    const CPP_TYPE: &'static str = "float";
    const SIZE: usize = size_of::<f32>();
}

#[repr(C)]
pub struct Metadata {
    magic: u64,
    type_info: &'static str,
    size: usize,
} 

pub fn emit_type_info<T: TypeInfo>(x: &T) {
    let m = Metadata {
        magic: MAGIC,
        type_info: T::CPP_TYPE,
        size: T::SIZE,
    };
    blackbox(&m); // can to external function - to make sure this is not optimized out
}

Each captured variable in a cpp! block would be expanded to emit_type_info(&var), and afterwards build script would search generated .rlib for metadata, in a similar manner as it does now with the lib generated by c++.

This would make build slower, of course, but the added convenience might be worth it...

[ogoffart]

What if build script used rustc itself to parse the crate?

Then it would only work with a nightly compiler, and never with stable.
So I don't think that is an option.

It is true that, in that case, we could use the type information from rustc to avoid redundant annotations.

We could do something similar as what you describe with the existing procedural macro. But then we would need a way to build the C++ after rust.
Cargo's current build.rs is always run before, i don't know if there are scripts we can run after.
Running the compiler once for every cpp! macro would also work, but i'd rather not as it would make the compilation really really slow. (maybe with pre-compiled header we can mitigate that, but still)

[vadimcn]

Then it would only work with a nightly compiler, and never with stable.

Why? I didn't mean to use libsyntax or anything like that, but rather spawn rustc as an external process.

We would need two versions of the cpp_macros crate:

• cpp_macros_pre, that saves cpp! snippets to a file and replaces them with calls to emit_type_info for each captured variable and the return type.
• cpp_macros, that is the same as current cpp_macros

The build sequence would be as follows:

1. cargo builds and spawns the build script
2. build script spawns rustc passing it --crate_type=rlib --crate-name=crate_pre --extern cpp_macros=cpp_macros_pre flags
3. rustc builds the crate and emits crate_pre.rlib
4. build script parses metadata from crate_pre.rlib, combines with c++ snippets, emits c++ source, invokes the c++ compiler.
5. cargo invokes rustc for the second time, pointing it to the "normal" cpp_macros, which works the same as today.

Obviously, this builds the crate twice, although if we didn't want to eliminate annotations, it could be sped up by not running codegen via --emit=metadata.

I am glossing over a lot of details, such as how to discover the rest of rustc command line parameters required to build crate_pre, but I figure this can be solved one way or another.

[luke-titley]

Hey. Was there any progress on this? I'm in the same situation at the moment.

[frankier]

I suppose there's not been any progress. Although I haven't used it myself, one thing to check out --- for comparisons sake if nothing else --- is Google's cxx and autocxx. These appear to have some kind of tools and approach for dealing with ownership between C++ and Rust.

[ogoffart]

@frankier: the cxx crate has the same problem.
It also uses a build script that parses the .rs files to extract some information before building the generated .cpp code with a C++ compiler. the #[cxx::bridge] module can't be generated from another macro.

[dtolnay]

the cxx crate has the same problem.

@ogoffart it doesn't, at least not in the context of #19 (comment). They are interested in using a macro to generate a CPlusPlusBox Rust type which is a new/delete smart pointer around a C++ MyCPlusPlusType. With the CXX library they'd idiomatically use cxx::UniquePtr<MyCPlusPlusType> and not need a macro in the first place.

[ogoffart]

This crate equivalent of CPlusPlusBox is to do

     cpp_class!(unsafe struct MyCPlusPlusType as "std::unique_ptr<MyCPlusPlusType>");

(that did not exist in 2017 when that comment was written)

But this issue is about embedding cpp! in other macros, so my comment was addressing that.
The ownership problem can be dealt with cpp_class! without the use of cpp! within macro.

[frankier]

Thanks to both of you for the info. It'll be very useful for next time I'm trying to wrap C++ with Rust. Sorry for derailing the issue!

[dtolnay]

I just mean that collecting why people reach for macros and solving those use cases in a better way is a viable way to make significant progress on this issue.

The ownership problem can be dealt with cpp_class! without the use of cpp! within macro.

I'm not convinced of this. A cpp_class with Default and Drop isn't all that goes into a smart pointer binding. You'd be back in macro land for something with as_ref, into_raw, new, supporting Deref to call methods on the inner type, Debug where the inner type implements Debug, etc.

[ogoffart]

true, cpp_class wrapping an unique_ptr is just box an opaque type, and there is no way to access the content. So this is usefull when one wants to expose a type that can't be move as its boxed variant, and add associated function on the boxed type itself to expose more functions of that type.

[HackerFoo]

I found a solution for this. I use cpprs to process C macros that generate cpp!(...) macros, which can then be processed by rust-cpp in the usual way.

It's not ideal, but it's better and more maintainable than cut-and-pasting. I'm tempted to write something that can expand a subset of Rust's macros with an annotation for this purpose, but this will do for now.