when use mass insert data to redis, loss data

[phenix3443]

i use python multiprocessing to insert a lot of recods to redis, every process use a method which is introduced in this article http://redis.io/topics/mass-insert, but finally the redis loss some data.

when total records is 50000,or 200000 , evey file have 10000 records, so ,there are 5 or 20 files. every process hanle one file. the data in redis is correct。

but when the totol record is 2000,000 . redis loss some data.

code as follow:
code.py.txt

[charsyam]

@phenix3443 How much memory do you have?
Is there evicted keys are in result of info commands?

[phenix3443]

@charsyam the redis use about 1.98G memory, i use one process to hanle all 200 file, but still loss data.

[charsyam]

@phenix3443 Do you check there are same keys?

[phenix3443]

there are not same key,wait a moment , i check the result every process return from redis-server.

[phenix3443]

@charsyam , the result from redis-server has something wrong, like
#2746
redis/node-redis#700

maybe i must solve these problem, but thank you very much.

[phenix3443]

@charsyam ,i find i can only store about 3G data in redis, use default config, then i can not store any more, the redis server disconnect the redis-cli --pipe connect. is there any suggestion?

[charsyam]

@phenix3443 how about using it on 64bit machine

[phenix3443]

this is 64bit machine,
uname -a
Linux twin145 2.6.18-308.el5 #1 SMP Tue Feb 21 20:06:06 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

[phenix3443]

i fond maxmemory set in configure files, may this cause the problem. i am trying to fix it.

[charsyam]

@phenix3443 Yes, If you already use it on 64bit, change maxmemory value or remove it.