How to insert multiple keys ?

[dardar4]

Hi
we need to implement multiple key insertion the fastest way is possible
I've read the following issue:
#432

but that didn't help me

what is the best practice to insert N <key, value> (where N > 1000 for example)
i hope that there is a better way than calling StringSet() N times

thanks

[mgravell]

The simplest way to do it would probably be to loop taking "some number" (100? 250? 500? - depends on the size of the keys and values, your network bandwidth, etc), and issue a number of MSET operations, which you can do via StringSet(KeyValuePair<RedisKey, RedisValue>[]). Perhaps, for example:

const int BatchSize = 100; // play with
var batch = new List<KeyValuePair<RedisKey, RedisValue>>(BatchSize);
foreach(var pair in yourSourceData)
{
    batch.Add(new KeyValuePair<RedisKey, RedisValue>(pair.Key, pair.Value));
    if (batch.Count == BatchSize) {
        db.StringSet(batch.ToArray());
        batch.Clear();
    }
}
if (batch.Count != 0) // final batch
  db.StringSet(batch.ToArray());

You may prefer to use await StringSetAsync, and/or CommandFlags.FireAndForget, depending on your needs. If you were using the async API, you could also defer each await until just before the next batch, so you continue building the next batch while the first processes, but that gets more complex.

At the extreme end: you could write a fixed-length scrolling pipe of awaitables and issue each item individually, but... I suspect that would actually have more overhead.

Any use?

[dardar4]

Hi
Thx for the detailed answer
Seems like a good solution , i will test it to see if this help

**EDIT:
the only thing that is a problem is the fact that the function overload that takes an array doesn't let you set an expiry time for some reason

bool StringSet(KeyValuePair<RedisKey, RedisValue>[] values, When when = When.Always, CommandFlags flags = CommandFlags.None);

[mgravell]

The "some reason" there is because: that isn't supported by redis itself - see MSET - it lacks any kind of expiry (the when parameter flips between MSET and MSETNX).

If you need timeouts as well, two options leap to mind:

• the pipelined async approach using SETEX (previously discussed)
• a Lua-based solution via EVAL/EVALSHA (ScriptEvaluate in SE.Redis)

Here's a full example of the second option that assumes a single expiry can be used for all the inserts:

using StackExchange.Redis;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;

static class P
{
    static void Main()
    {
        // WARNING: example flushes database as part of execution

        const int DB = 0;
        var muxer = ConnectionMultiplexer.Connect(new ConfigurationOptions {
            EndPoints = { "127.0.0.1" },
            AllowAdmin = true, // enables FLUSHDB for our example
        });
        var db = muxer.GetDatabase(DB);
        var server = muxer.GetServer(muxer.GetEndPoints().Single());

        Console.WriteLine($"FLUSHING DATABASE {DB}...");
        server.FlushDatabase(DB);

        Console.WriteLine($"size before: {server.DatabaseSize(DB)}");

        const int BatchSize = 100, ExpirySeconds = 120;
        List<RedisKey> keys = new List<RedisKey>(BatchSize);
        List<RedisValue> values = new List<RedisValue>(BatchSize + 1);

        // note that ARGV[1] is the expiry, so all the values are
        // off-by-one compared to their keys
        const string lua = @"
local expiry = tonumber(ARGV[1])
local count = 0
for i, key in ipairs(KEYS) do
  redis.call('SETEX', key, expiry, ARGV[i+1])
  count = count + 1
end
return count
";
        values.Add(ExpirySeconds);
        foreach(var pair in InventData(1024))
        {
            keys.Add(pair.key);
            values.Add(pair.value);

            if(keys.Count == BatchSize)
            {
                // execute
                Console.WriteLine($"sending batch of {keys.Count}...");
                var count = (int)db.ScriptEvaluate(lua, keys.ToArray(), values.ToArray());
                Debug.Assert(count == keys.Count); // check expectation

                // reset for next batch
                keys.Clear();
                values.Clear();
                values.Add(ExpirySeconds);
            }
        }
        if (keys.Count != 0)
        {
            // execute final batch
            Console.WriteLine($"sending batch of {keys.Count}...");
            var count = (int)db.ScriptEvaluate(lua, keys.ToArray(), values.ToArray());
            Debug.Assert(count == keys.Count); // check expectation
        }

        Console.WriteLine($"size after: {server.DatabaseSize(DB)}");
    }

    static IEnumerable<(string key, string value)>
        InventData(int count)
    {
        var rand = new Random();
        unsafe string Invent(int len)
        {
            string alphabet = "0123456789 abcdefghijklmnopqrstuvwxyz";
            string s = new string('\0', len);
            fixed(char* ptr = s)
            {
                for (int i = 0; i < len; i++)
                    ptr[i] = alphabet[rand.Next(alphabet.Length)];
            }
            return s;
        }
        for(int i = 0; i < count; i++)
        {
            yield return (Invent(20), Invent(50));
        }
    }
}

[dardar4]

thanks alot
both solutions seems valid to me. we'll give it a try

[ilmam]

@mgravell that's a great solution for Lua script. Is there any way we can keep all the batches in one transaction? So that any batch fails rolls back the the whole transaction?

[mgravell]

Redis is not that kind of database, so: no