Feature Request: Batch Insert #255
Comments
|
on the same point, batch row updates would be awesome too. |
|
I have thought about it before, but haven't implement it due to can't run callbacks. |
|
👍 |
|
+1 for Batch insert |
|
Is there still no way to accomplish this? I tried building a string of multiple queries and passed it to Exec or Raw but I get errors when using Exec. Raw just seems not to do anything. the actual sql works fine in mysql
for now I'll just use individual statements |
|
+1 |
|
the problem is in the mysql db driver. |
|
+1, any way to do it now? |
|
👍 |
|
Waiting for this feature eagerly aswell. |
|
👍 |
|
+1 for Batch Inserts |
|
great news guys! |
In the handler binding, remove the macaron's data structure binding,
since it cannot support []interface{} for now.
In the handler, since the req param is an array, insert the batch data
into database at once by a raw SQL, since gorm currently doesn't support
create multiple records at once. It's a missing feature by the mysql-driver,
which is discussed here:
https://github.com/jinzhu/gorm/issues/255
|
One of the things I'd really like is to be able to INSERT many values at once in a single query (for speed):
... etc. I realize there are some issues here with the fact that (for example) the PrimaryKey/ID wouldn't be returned for multiple rows in this case, and this would require a subsequent SELECT to get that/those. However, that could also just be a disclaimer when using this function in the documentation. Would be nice to be able to pass |
|
What do we need to do to start moving on this? I'd like to use gorm for a project but this is a blocker for me. It seems Like we'd need to trigger the callbacks once for each object in the slice. The tricky part would be error handling and reporting which could either fail immediately or continue processing on errors. |
|
@TylerBrock It's something you can work around by constructing the queries manually, but that kind of sucks. |
|
@benguild what would that look like? I'm just starting out with Golang and Gorm but maybe providing a workaround solution example would help a lot of people in this thread out. Would you mind showing us how something like that would be done? Is it just a single transaction that executes multiple inserts via Gorm or would you use the pg driver directly? |
|
@TylerBrock Something like this: https://github.com/jinzhu/gorm/issues/920 (except for batch insert: http://stackoverflow.com/questions/5526917/how-to-do-a-batch-insert-in-mysql ) |
|
Yup of course, thanks @pjebs, the SQL syntax I don't have any questions about. What I was hoping was to see was something where gorm's helpers can be used to construct the statement in a more programmatic fashion but basically the Raw command and SQL builder part of the docs cover it, thanks! |
|
Yeah dropping down to raw SQL was what I meant. Just be aware that as I
mentioned the last INSERT ID won't be filled in for the models after INSERT
unless you do a manual batch SELECT thereafter. Not doing that could create
some undesirable, unexpected, or suboptimal behavior... which I'm guessing
is why this hasn't been tackled yet.
However it's incredibly suboptimal to run 50 queries instead of 1, for
example, so running a SELECT afterward isn't as big a deal in some cases. I
think the logic I used to handle these cases was to only use the batch
INSERT/SELECT format if at least 3 rows needed to be created. Otherwise it
just did two solo INSERTs if a SELECT would need to take place afterward to
get the IDs... since the solo INSERTs return them automatically.
--
iPhoneから送信
|
|
Is Wouldn't you want to |
|
@sandalwing Yes it is a type... that's what I said here: https://github.com/jinzhu/gorm/issues/255#issuecomment-243997661
... So yeah, that would be a way to do it. However, a |
|
+1 |
|
If anyone ends up here, I had a great experience by integrating https://github.com/t-tiger/gorm-bulk-insert with gorm. |
|
Please implement bulk functions for common queries, including insert, update, and delete. In medium to large applications, batching becomes a first tier priority for maintaining efficient, scalable systems. |
|
Just to remind you guys: don't forget to support |
|
+1 |
|
@jinzhu Do you appreciate the +1 or do you keep track of the votes in the original post? I would really like this and I can help too so I’m watching this issue but I’m kind of against all the +1. If it’s not the desired way according to the maintainers could we try to do a 👍 on the original post instead? EDIT: I made gorm-bulk similar to the comments above and the one mentioned by t-tiger but with a (in my opinion) more flexible approach letting the end user define their own scope (SQL, and values). It also help you generate code to convert to the required interface slice. It has three default bulk actions bundled as of now (instead of just |
|
Hey, |
|
Wow! This feature originally open in 2014, +1ed by soooo many and still not closed. Whats stopping this? |
|
+1 for batch insert. When I tried to put []interface{} as parameter to db.Save and get back this panic "reflect: call of reflect.Value.Interface on zero Value" |
|
Added its support in v2, so going to close this loooooong issue. |
|
Hey, @jinzhu thanks for this. Is v2 ready for use? |
|
@King-Success it's on a branch: https://github.com/jinzhu/gorm/tree/v2_dev pull it down and give it a whirl! |
|
Thanks, @eldilibra |
Hello! which function does the bulk insert in this branch? thanks! |
It's the regular methods you use for single row inserts. Just pass a slice of your type and type MyType struct {
Value string
}
func main() {
// ...
toInsert := []MyType{
{Value: "first row"},
{Value: "second row"},
{Value: "third row"},
}
db.Create(toInsert)
}Will create a query similar to INSERT INTO "my_types" ("value") VALUES ('first row'),('second row'),('third row') |
|
Hello I can successfully use this func to insert single value : But when I use this, it's become error Anyone can help? thanks |
|
This is what I did: given these structs type Item struct {
Name string
}
type InputItem struct {
Name string
}part that bulk inserts a slice var input []InputItem
var items []Item
tx := db.Begin()
defer func() {
if r := recover(); r != nil {
tx.Rollback()
}
}()
if txErr := tx.Error; txErr != nil {
panic(txErr)
}
for _, i := range input {
item := Item{Name: i.Name}
items = append(items, item)
if createErr := tx.Create(&item).Error; createErr != nil {
panic(createErr)
}
}
if commitErr := tx.Commit().Error; commitErr != nil {
panic(commitErr)
} |
|
how to implement bulk insert with ignore??? like |
|
@easy-money-sniper For v1, have a look at gorm-bulk and the feature BulkInsertIgnore(). |
|
Is it possible to scan back the ids of the bulk inserted records (like in case of a single record)? I tried Scan(&[]recordType{}) and ScanRows(), but they doesn't seem to work ("throws reflection error, "comparing with nil"). |
@dark-shade Yes, will scan back ids
db.Clauses(clause.OnConflict{DoNothing: true}).Create(&tasks) |
and others
pass in []interface{} to batch insert using db.Create([]interface{})
The text was updated successfully, but these errors were encountered: