Using unsupported buffer type: 253 (parameter: 1) when trying to use the CLI

[watsom27]

Setup

• Latest version of diesel_cli
• MySql 8.3.0 running in docker

Versions

• Rust: rustc 1.75.0 (82e1608df 2023-12-21)
• Diesel: 2.1.1
• Database: MySql 8.3.0
• Operating System MacOs

Feature Flags

• diesel: mysql

Problem Description

When running any commands using the diesel command, the error below is given:

$: diesel migration list
Using unsupported buffer type: 253  (parameter: 1)

Its worth noting that diesel setup does create the database, but fails to run any migrations.

What are you trying to accomplish?

Running migrations

What is the expected output?

Migrations that run? Not sure what else to put here

What is the actual output?

$: diesel setup
Creating migrations directory at: [my_directories]/migrations
Creating database: DieselTest
Failed to run migrations: Using unsupported buffer type: 253  (parameter: 1)

Are you seeing any additional errors?

No

Steps to reproduce

Repo: https://github.com/watsom27/diesel_bug

1. Spin up the MySql docker

docker compose up

2. Try diesel command

diesel setup

Checklist

• I have already looked over the issue tracker and the discussion forum for similar possible closed issues.

• This issue can be reproduced on Rust's stable channel. (Your issue will be
  closed if this is not the case)
• This issue can be reproduced without requiring a third party crate

[watsom27]

As always happens, just after opening this issue i have managed to solve it 🤦🏻

I'm assuming this is a compatability issue between Diesel and MySql 8.3.

I solved this by:

1. Installing the previous version of mysql using brew install mysql@8.0
2. Locate the directory containing the libmysqlclient files. For me this was /opt/homebrew/Cellar/mysql\@8.0/8.0.36/lib/
3. Install diesel_cli and point it at that directory for the client lib: MYSQLCLIENT_LIB_DIR=/opt/homebrew/Cellar/mysql\@8.0/8.0.36/lib/ cargo install diesel_cli --no-default-features --features=mysql --force

[weiznich]

Thanks for opening this issue and adding these details. I would say it is expected that diesel works with the latest libmysqlclient version, therefore that's likely a bug. So I reopen this issue.

Can you verify a detail: You can fix this by just using a older libmysqlclient version with the newer database? (So client 8.0, database 8.3)

[watsom27]

Can you verify a detail: You can fix this by just using a older libmysqlclient version with the newer database? (So client 8.0, database 8.3)

Correct, the 8.0 libmysqlclient seems to work with the 8.3 MySql docker image. Although I haven't done a full in depth test, but it creates the database and runs the migrations.

[alexwatever]

I also faced this issue when re-installing MySQL on MacOS. I first saw the below overflow/underflow error when querying a table that had not been touched in a long time:

DeserializationError("Numeric overflow/underflow occurred")

And finally came across the "Using unsupported buffer type: 253 (parameter: 1)" error when trying to list migrations, which pointed me to this issue.

I had to run cargo clean but your solution did the trick:

brew remove mysql
brew install mysql@8.0
MYSQLCLIENT_LIB_DIR=/opt/homebrew/Cellar/mysql\@8.0/8.0.36/lib/
cargo clean
cargo install diesel_cli --version 2.1.1 --no-default-features --features "mysql" --force
diesel migration list

Thank you @watsom27 you're a lifesaver - this was doing my head in!

[weiznich]

That really sounds like something changed fundamentally in the mysql client headers at the c side. In turn that means we likely need to update the generated bindings for that + adjust some code in diesel. Unfortunately I do currently not have the capacity to do all of that on my own, so I need to mark this as accepting contributions and let it sit here till someone spends the time to perform the necessary steps. (Given that this seems to happen on a minor mysql release I highly question their approach of handling breaking changes…)

[watsom27]

@weiznich I would be happy to have a go at this, but having looked at the codebase I have no idea where to start.

If you could point me in the general direction of the code this change would involve, or if there is any developer documentation I will happily take a look.

Thanks

[weiznich]

@watsom27 It's totally fine to ask for help where to start and what to try. I know that the codebase is large and complex, so you are not expected to solve this completely on your own.

I would likely start by running at these tests with the new libmysqlclient version.

diesel/diesel/src/mysql/connection/bind.rs

Line 736 in ad8ef47

mod tests {

Given the error message in the bug report I would expect that just running them will produce some similar error. That should give use more information about which type is affected. Starting from there I would likely check the header files of the new and old libmysqlclient version to check if the relevant bind type for that sql type has changed. Maybe add some println statements in there and see how the values change between versions. If we have more information about which type is affected and why that happens we can discuss ways how to solve that without breaking compatibility with the old versions.

[clouds56]

Share some investigation here (note: the line number might offset by 2-3 lines since I added some debug print).
last_error_number 2036

for test in tests::queryable_by_name::struct_with_no_table (at ./tests/queryable_by_name.rs:70:16)

// failed on this query
sql_query("SELECT 1 AS foo, 2 AS bar").get_result(conn);

downside I've tracked down to StatementIterator, which might be corrupt as early as

// the metadata returned by this api is not accessible from rust
let metadata = stmt.metadata(); // underlying delegate to `ffi::mysql_stmt_result_metadata`
// when trying to print fields of underlying raw pointer
println!("field_count: {}", (&unsafe { *result_ptr }).field_count); // this should be `2`, but `0` printed.
// when trying to print all fields via, segment faults with `(signal: 11, SIGSEGV: invalid memory reference)`
println!("fields: {:?}", metadata.fields());

by comparing st_mysql_res in mysqlclient-sys-0.2.5, I've add some comment to help alignment.

pub struct st_mysql_res {
    // uint64_t row_count;
    pub row_count: my_ulonglong,
    // MYSQL_FIELD *fields;
    pub fields: *mut MYSQL_FIELD,
    // struct MYSQL_DATA *data;
    pub data: *mut MYSQL_DATA,
    // MYSQL_ROWS *data_cursor;
    pub data_cursor: *mut MYSQL_ROWS,
    // unsigned long *lengths; /* column lengths of current row */
    pub lengths: *mut ::std::os::raw::c_ulong,
    // MYSQL *handle;          /* for unbuffered reads */
    pub handle: *mut MYSQL,
    // const struct MYSQL_METHODS *methods;
    pub methods: *const st_mysql_methods,
    // MYSQL_ROW row;         /* If unbuffered read */
    pub row: MYSQL_ROW,
    // MYSQL_ROW current_row; /* buffer to current row */
    pub current_row: MYSQL_ROW,
    // struct MEM_ROOT *field_alloc;
    pub field_alloc: MEM_ROOT, // This field is embedded in rust
    // unsigned int field_count, current_field;
    pub field_count: ::std::os::raw::c_uint,
    pub current_field: ::std::os::raw::c_uint,
    // bool eof; /* Used by mysql_fetch_row */
    pub eof: my_bool,
    // bool unbuffered_fetch_cancelled;
    pub unbuffered_fetch_cancelled: my_bool,
    // enum enum_resultset_metadata metadata;
    // void *extension;
    pub extension: *mut ::std::os::raw::c_void,
}

and corresponding C struct (from mysql master, 8.3.0)

typedef struct MYSQL_RES {
  uint64_t row_count;
  MYSQL_FIELD *fields;
  struct MYSQL_DATA *data;
  MYSQL_ROWS *data_cursor;
  unsigned long *lengths; /* column lengths of current row */
  MYSQL *handle;          /* for unbuffered reads */
  const struct MYSQL_METHODS *methods;
  MYSQL_ROW row;         /* If unbuffered read */
  MYSQL_ROW current_row; /* buffer to current row */
  // ------------------------
  // NOTE here in C struct, the MEM_ROOT is a pointer
  // ------------------------
  struct MEM_ROOT *field_alloc;
  unsigned int field_count, current_field;
  bool eof; /* Used by mysql_fetch_row */
  /* mysql_stmt_close() had to cancel this result */
  bool unbuffered_fetch_cancelled;
  // ------------------------
  // NOTE here in C struct, a new field not in `st_mysql_res`
  // ------------------------
  enum enum_resultset_metadata metadata;
  void *extension;
} MYSQL_RES;

Note that the difference in alignment of MYSQL_RES is not critical since we use ffi::mysql_num_fields and ffi::mysql_fetch_fields to access fields.

diesel/diesel/src/mysql/connection/stmt/metadata.rs

Lines 18 to 28 in a5c82df

	pub(in crate::mysql::connection) fn fields(&'_ self) -> &'_ [MysqlFieldMetadata<'_>] {
	unsafe {
	let num_fields = ffi::mysql_num_fields(self.result.as_ptr());
	let field_ptr = ffi::mysql_fetch_fields(self.result.as_ptr());
	if field_ptr.is_null() {
	&[]
	} else {
	slice::from_raw_parts(field_ptr as _, num_fields as usize)
	}
	}
	}

But the difference (size) between st_mysql_field and MYSQL_FIELD would cause the error since we treat the return value as an array of MysqlFieldMetadata, or say &[st_mysql_field].

pub struct st_mysql_field {
    // char *name;
    pub name: *mut ::std::os::raw::c_char,
    // char *org_name;
    pub org_name: *mut ::std::os::raw::c_char,
    // char *table;
    pub table: *mut ::std::os::raw::c_char,
    // char *org_table;
    pub org_table: *mut ::std::os::raw::c_char,
    // char *db;
    pub db: *mut ::std::os::raw::c_char,
    // char *catalog;
    pub catalog: *mut ::std::os::raw::c_char,
    // /* THIS FIELD IS NOT PRESENT IN C STRUCT */
    pub def: *mut ::std::os::raw::c_char,
    // unsigned long length;
    pub length: ::std::os::raw::c_ulong,
    // unsigned long max_length;
    pub max_length: ::std::os::raw::c_ulong,
    // unsigned int name_length;
    pub name_length: ::std::os::raw::c_uint,
    // unsigned int org_name_length;
    pub org_name_length: ::std::os::raw::c_uint,
    // unsigned int table_length;
    pub table_length: ::std::os::raw::c_uint,
    // unsigned int org_table_length;
    pub org_table_length: ::std::os::raw::c_uint,
    // unsigned int db_length;
    pub db_length: ::std::os::raw::c_uint,
    // unsigned int catalog_length;
    pub catalog_length: ::std::os::raw::c_uint,
    // /* THIS FIELD IS NOT PRESENT IN C STRUCT */
    pub def_length: ::std::os::raw::c_uint,
    // unsigned int flags;
    pub flags: ::std::os::raw::c_uint,
    // unsigned int decimals;
    pub decimals: ::std::os::raw::c_uint,
    // unsigned int charsetnr;
    pub charsetnr: ::std::os::raw::c_uint,
    // enum enum_field_types type;
    pub type_: enum_field_types,
    // void *extension;
    pub extension: *mut ::std::os::raw::c_void,
}

I still can't see how specifying MYSQLCLIENT_LIB_DIR would bypass this.

[clouds56]

Maybe we should updating bindings for mysqlclient-sys, sgrif/mysqlclient-sys#24,
or not use a fix binding, instead we could generate one on build.

[weiznich]

@clouds56 Thanks for investigating. That's really helpful ❤️

I've marked this as release blocker now, as we should resolve this before issuing a new release.

Given the current state of the mysqlclient-sys crate I would expect that this is unfortunately a bit more work than "just" updating the bindings.
We should likely setup a CI there as first step. After that we likely need to check if the new bindings also work with older versions otherwise that would be problematic.

I'm generally not a big fan of only generating the bindings at build time as that requires a dependency on bindgen, which in turn requires libclang, which might be problematic on windows.

(Help by all of this is appreciated.)

[xiangzhai]

Hi Apple fans,

Thanks for @clouds56 investigation.

mysql 8.0.1 Remove my_bool and then added MYSQL_TYPE_INVALID, MYSQL_TYPE_BOOL and MYSQL_TYPE_TYPED_ARRAY fields for enum_field_types, so I just change mysql connection for macos mysqlclient 8.3.

And I also use rust-bindgen to autogen the bindings.rs for macos, so there is no need to maintain the bindings_os.rs any more.

Thanks,
Leslie Zhai

[weiznich]

@xiangzhai Thanks for looking into this. This is likely a good starting point, but I fear it won't be as easy as just adjusting the code to work with the new version (only) as we also need to support older versions. I'm not sure yet how to do that.

As for auto generating the bindings: I would consider accepting a PR to mysqlclient-sys that adds that behind a feature flag. I do not want to have that enabled by default as it requires libclang to be present on the build system, which might not be the case on most windows systems.

[xiangzhai]

Hi @weiznich

You are welcome!

I am considering about LIBMYSQL_VERSION and MYSQL_VERSION_ID to get the VERSION_ID and also check whether or not it is MySQL or MariaDB via MYSQL_SERVER_VERSION in the mysqlclient-sys side. Then it is able to cover the new version and also support the older versions:

let (version_id, is_mysql, is_mariadb) = ffi::get_libmysql_version_id();
assert!(0 != version_id, "Invalid MySQL Version ID");

if is_mysql {
...
}

if is_mariadb {
...
}

I just wrapper the my_bool and mysql_ssl_mode because there is no mysql_ssl_mode for MariaDB. Then it is able to still use mysql_ssl_mode in diesel side:

use mysqlclient_sys::mysql_ssl_mode;
...

But there is ugly FALSE to support my_bool. so how to fix it :P

#[cfg(target_os = "linux")]
static FALSE: ffi::my_bool = 0;
#[cfg(target_os = "macos")]
static FALSE: ffi::my_bool = false;

Could you review the path to give some comments please?

Thanks,
Leslie Zhai

[xiangzhai]

it requires libclang to be present on the build system, which might not be the case on most windows systems.

Add windows support

Thanks,
Leslie Zhai