Replication error - Database error: ERROR: tuple concurrently deleted

[mwierzbi]

Describe the bug
When i try to update database i have got the error:
ERROR: DB copy thread failed: Database error: ERROR: tuple concurrently deleted

To Reproduce
Steps to reproduce the behavior:

sudo -u nominatim nominatim replication -j 5 --check-for-updates
2023-12-05 13:04:58: Using project directory: /nominatim
2023-12-05 13:04:58: New data available (3829 => 3901).
root@7ac33cd3b104:/nominatim# sudo -u nominatim nominatim replication -j 5 --once             
2023-12-05 13:05:51: Using project directory: /nominatim
2023-12-05 13:06:06  osm2pgsql version 1.7.2
2023-12-05 13:06:06  Database version: 14.9 (Ubuntu 14.9-0ubuntu0.22.04.1)
2023-12-05 13:06:06  PostGIS version: 3.2
2023-12-05 13:06:06  Parsing gazetteer style file '/usr/local/etc/nominatim/import-full.style'.
Processing: Node(699k 349.5k/s) Way(159k 39.75k/s) Relation(1500 750.0/s)
2023-12-05 13:06:14  ERROR: DB copy thread failed: Database error: ERROR:  tuple concurrently deleted

Traceback (most recent call last):
  File "/usr/local/bin/nominatim", line 14, in <module>
    exit(cli.nominatim(module_dir='/usr/local/lib/nominatim/module',
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 264, in nominatim
    return parser.run(**kwargs)
  File "/usr/local/lib/nominatim/lib-python/nominatim/cli.py", line 126, in run
    return args.command.run(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 196, in run
    self._update(args)
  File "/usr/local/lib/nominatim/lib-python/nominatim/clicmd/replication.py", line 153, in _update
    state = replication.update(conn, params, socket_timeout=args.socket_timeout)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/replication.py", line 133, in update
    run_osm2pgsql_updates(conn, options)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/replication.py", line 154, in run_osm2pgsql_updates
    run_osm2pgsql(options)
  File "/usr/local/lib/nominatim/lib-python/nominatim/tools/exec_utils.py", line 159, in run_osm2pgsql
    subprocess.run(cmd, cwd=options.get('cwd', '.'),
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/lib/nominatim/osm2pgsql', '--hstore', '--latlon', '--slim', '--log-progress', 'true', '--number-processes', '1', '--cache', '2000', '--style', '/usr/local/etc/nominatim/import-full.style', '--output', 'gazetteer', '--append', '--flat-nodes', '/nominatim/flatnode/flatnode.file', '--with-forward-dependencies', 'false', '/nominatim/osmosischange.osc']' returned non-zero exit status 2.

Screenshots & Logs

2023-12-05 13:06:14.201 UTC [192736] nominatim@nominatim ERROR:  tuple concurrently deleted
2023-12-05 13:06:14.201 UTC [192736] nominatim@nominatim STATEMENT:  DELETE FROM "planet_osm_rels" WHERE id IN (4747,5565,6141,8029,10692,14949,14997,16433,17436,17760,18524,20828,21108,21563,21915,22283,22487,22679,23088,23502,23552,25571,26661,26702,26821,27479,27485,27962,28057,29180,30069,30449,31045,31580,31626,32623,33351,33555,34122,35040,35983,36980,37024,37247,37937,38267,38272,38303,38443,38625,38837,48572,48800,50257,51267,51295,52230,52947,52950,53638,54457,57198,57199,57216,57217,59413,59622,60189,60199,60433,61396,61491,62341,62555,62587,62615,62761,62986,62987,63464,65725,65855,66003,66650,68615,68752,69122,70682,71918,71971,72022,74824,75260,75473,75721,77227,77964,79018,82569,83383,85093,85101,85302,85303,85606,86277,86485,87033,88498,89261,89691,91631,102470,104397,104742,107031,107742,108349,121088,122709,123788,123822,124706,124889,124913,125001,125452,127021,129297,129897,130768,135833,136286,136376,138737,140777,140857,140922,142867,143147,143748,144173,148540,149865,150163,151754,153521,153606,154766,158156,158479,161236,164332,164333,164517,165923,168363,171838,171846,175374,177082,177561,177588,177745,179724,181787,186493,187700,188538,189406,190244,191725,192649,195443,197026,199892,203440,215762,216246,222560,223462,223473,225164,225166,225469,228027,228600,233160,233172,233711,236930,238704,239279,254682,254692,255517,269427,269905,270230,270235,270236,270285,274286,275649,278940,281776,281780,281858,282665,283891,284323,284631,284652,285035,286530,288686,289164,290086,296432,300605,301054,303198,308326,309577,309599,311474,317609,318280,319019,324934,326757,331996,333125,334336,334377,334972,335265,335268,335450,337465,354405,357628,358784,359070,360048,360049,363581,365969,365987,366495,366496,366608,367817,372666,372701,374438,376424,376758,378304,378305,379554,384412,384642,385004,387323,388619,388773,388880,388963,390299,390306,390375,395707,399343,401629,401713,402520,403107,403186,404698,404929,405857,405858,405859,405860,405874,405875,407955,408362,409021,409150,409354,409541,410724,410763,411461,415290,416224,416720,416759,416771,416782,416786,416815,416971,417178,417966,418004,418090,418282,418284,418354,421299,421321,422093,422651,422882,422884,423204,441776,444489,447530,448725,450476,450535,450536,454974,536511,539017,542312,544354,544622,545048,548465,548790,549426,549429,554601,569064,570895,571022,572248,660767,660775,812312,898965,905348,905629,905631,905632,905641,905645,905655,906813,908805,915656,915672,916959,916960,917537,917958,918111,918112,919017,919322,919362,919364,921619,929371,929382,929532,929575,936729,937188,937194,937201,937303,939570,940084,943686,944475,944892,945489,949814,949820,951739,957129,959114,963212,976120,976300,977327,983433,983587,983642,983740,984009,984078,1005939,1007288,1008903,1009693,1010949,1011121,1014387,1016369,1021034,1026555,1026599,1026601,1059500,1075100,1075102,1076003,1084578,1084583,1101171,1101702,1102468,1102684,1102685,1107741,1113178,1113220,1113222,1117508,1117520,1119047,1124734,1124871,1125460,1125716,1125717,1125718,1127217,1127886,1128902,1130286,1135463,1138921,1140299,1145937,1147282,1148166,1148170,1148322,1153801,1154537,1158484,1162695,1164003,1164020,1164179,1164200,1164206,1164218,1165305,1169183,1171191,1174166,1174175,1176276,1176279,1180900,1180901,1180902,1180924,1180925,1180976,1180977,1181383,1185182,1185184,1187076,1189601,1189602,1189606,1189610,1197592,1202524,1202525,1202528,1205688,1205872,1206975,1207838,1210026,1213907,1214085,1216886,1222833,1224330,1224621,1231495,1231708,1231710,1235973,1242417,1245522,1245648,1245766,1247020,1248824,1249581,1249583,1249599,1249600,1251260,1252498,1259848,1262955,1266664,1269027,1269797,1270058,1270060,1270065,1277694,1280704,1280705,1281044,1281045,1281046,1281047,1283096,1284257,1284275,1285127,1285423,1285664,1285674,1288310,1289560,1290087,1290089,1303671,1305676,1307231,1312559,1317290,1325788,1325789,1330044,1334220,1334221,1334222,1334223,1334997,1334998,1336030,1336394,1338486,1342786,1344024,1346265,1352277,1352888,1358506,1361900,1368644,1368645,1369386,1370699,1370700,1378986,1379199,1379200,1380084,1385331,1389578,1390897,1391528,1400257,1400388,1403786,1405516,1407665,1421038,1421089,1423225,1427143,1432612,1441227,1441256,1445698,1446222,1446247,1446261,1447852,1448569,1450020,1452047,1459232,1459233,1459416,1468300,1468396,1471005,1473354,1476090,1478343,1480165,1485435,1490894,1491830,1505866,1507634,1507635,1513204,1514375,1516909,1516929,1523367,1523510,1523518,1525086,1527326,1527379,1530590,1532357,1535976,1536162,1536725,1544124,1544129,1546757,1546758,1546876,1546877,1546900,1546902,1546931,1546979,1546983,1546991,1546997,1549274,1550598,1551290,1553173,1554211,1554477,1555815,1558971,1559389,1559394,1559396,1559399,1559405,1559408,1559409,1559875,1561757,1561761,1561762,1566572,1570737,1570738,1571297,1572005,1572436,1578618,1578642,1578881,1578882,1584422,1586269,1587283,1587449,1589514,1589802,1595465,1598979,1598981,1598985,1599012,1599014,1601011,1601240,1601262,1601531,1604587,1605651,1607390,1607477,1607956,1607957,1607958,1608349,1610943,1618715,1620897,1621860,1628200,1628201,1643603,1650088,1652207,1655777,1656086,1657684,1658342,1659976,1667201,1668440,1669461,1669462,1669733,1672529,1675428,1676283,1685895,1688177,1688178,1688179,1688180,1692834,1696127,1696142,1698598,1701305,1702323,1708757,1708763,1718785,1718788,1726774,1729062,1729064,1730102,1731854,1732488,1733560,1737352,1737354,1741202,1747395,1750241,1751373,1751600,1751601,1752165,1752875,1757153,1757154,1762085,1762616,1763003,1764005,1764594,1770442,1770651,1773963,1778714,1782624,1783472,1784834,1789940,1789941,1791711,1794173,1794174,1795678,1797113,1800293,1811128,1812355,1819043,1819916,1821136,1821607,1827690,1830560,1831340,1835722,1836516,1840315,1841866,1842944,1848421,1848430,1853359,1853360,1856706,1861853,1864043,1875141,1875285,1875338,1875340,1876153,1879606,1881807,1889430,1894817,1895662,1900139,1901599,1901600,1901946,1904106,1905347,1905348,1905349,1907541,1909599,1910414,1911568,1912909,1914523,1914621,1915264,1922071,1930228,1930229,1930230,1930231,1930233,1930234,1939030,1939031,1940856,1946367,1950786,1950931,1952915,1968651,1968652,1971920,1988463,1990539,1995719,1995932,1995933,1998795,1999275,2000662,2003484,2003667,2004451,2006258,2007026,2013834,2016358,2016359,2016361,2018554,2026548,2026607,2045294,2048360,2048535,2052881,2053854,2053866,2055374,2058686,2060610,2060792,2061680,2063075,2064835,2066482,2067385,2071399,2073122,2074057,2074058,2075336,2077646,2077981,2078079,2078341,2079657,2082967,2084109,2084473,2084497,2087590,2096486,2100951,2102260,2102794,2102795,2104741,2106574,2108123,2108267,2114147,2116473,2126744,2126849,2127114,2128459,2137039,2139913,2140101,2140115,2141287,2142136,2145264,2145266,2145908,2146222,2147280,2148629,2150835,2151556,2152379,2152490,2155121,2156366,2162786,2163561,2166895,2166898,2166928,2166929,2166961,2168335,2168336,2172341,2175381,2178900,2179272,2181514,2181515,2181516,2181517,2181518,2181519,2182237,2183030,2183431,2183433,2183590,2185147,2188577,2188872,2191579,2192204,2194813,2195417,2196907,2203608,2209527,2210491,2213848,2215085,2215237,2219219,2219220,2221551,2221907,2221928,2229107,2232290,2232622,2235992,2236017,2236018,2238556,2245558,2246156,2248443,2248444,2253881,2254187,2254966,2255855,2256131,2256207,2256226,2256747,2257315,2259418,2260699,2265095,2272060,2277158,2279215,2281177,2283779,2283808,2297972,2298003,2298724,2305212,2305213,2306544,2307304,2307305,2307611,2307788,2310624,2312832,2314106,2315316,2316342,2316372,2317706,2321658,2321888,2326665,2333761,2340973,2341020,2341273,2342188,2345955,2345960,2345965,2345971,2345975,2349927,2350597,2350756,2354910,2357552,2361906,2369861,2372136,2378354,2382745,2382746,2382747,2382748,2383000,2384373,2390718,2393946,2394035,2394076,2394194,2394197,2394201,2394205,2394217,2394372,2394373,2396447,2398804,2399124,2405631,2406462,2412751,2420166,2424538,2424539,2425832,2425833,2428019,2429201,2431526,2432227,2432228,2432229,2434144,2434146,2438730,2439356,2441589,2442868,2456723,2462979,2463076,2464720,2464722,2464723,2465033,2465066,2465075,2470258,2472328,2497734,2502145,2503075,2503619,2503684,2504274,2504566,2505051,2505057,2505060,2505482,2505485,2512807,2516662,2517396,2517397,2517398,2517399,2517447,2530104,2533504,2551637,2562291,2564071,2565564,2569042,2572958,2574596,2574937,2575593,2578431,2579144,2579637,2589367,2604306,2611805,2616298,2616413,2624116,2624119,2625573,2629521,2631043,2631944,2631945,2632075,2633139,2633172,2639185,2640262,2642817,2659443,2661482,2665208,2665209,2665227,2667657,2667902,2674708,2676451,2678040,2681798,2681848,2681849,2681861,2681866,2681874,2681902,2681906,2681976,2682003,2682008,2682016,2682024,2682025,2682028,2682057,2682073,2682075,2682103,2682105,2682106,2682107,2682135,2682140,2682142,2682144,2682149,2682153,2682154,2682181,2682194,2682366,2684370,2689577,2690662,2691196,2691197,2698636,2699560,2699561,2701743,2702468,2704631,2705408,2710108,2710188,2726289,2729649,2733743,2737901,2737902,2737904,2737905,2737906,2737907,2738621,2738622,2738623,2743167,2743625,2746502,2752454,2756278,2764897,2766945,2773183,2777909,2781040,2781736,2783608,2783610,2785111,2788485,2793232,2793464,2798624,2800304,2800319,2804477,2808006,2808030,2812416,2812417,2825365,2825366,2829991,2835484,2838632,2839216,2850014,2850015,2857784,2858680,2858681,2870931,2874857,2877495,2879897,2880892,2887767,2890293,2892490,2893143,2893419,2894192,2896213,2896247,2896248,2898429,2900845,2900846,2903869,2903870,2903873,2903874,2911526,2914483,2917919,2921164,2923575,2923578,2923582,2923583,2923584,2923585,2923586,2923587,2923588,2923589,2925468,2925918,2929015,2929040,2929186,2929615,2934371,2936494,2936562,2952565,2952566,2952567,2957020,2957374,2964926,2978260,2981797,2981798,2983228,2985520,3004553,3004555,3004631,3005135,3005136,3005137,3005138,3005139,3005140,3005141,3005142,3005143,3009860,3011271,3012031,3012320,3012877,3015734,3015735,3020769,3020770,3020801,3020802,3022413,3023421,3023770,3023930,3030049,3048924,3049322,3049425,3049511,3049512,3059810,3059910,3062192,3066924,3069302,3074941,3074942,3076745,3077747,3081493,3086807,3089653,3091791,3091881,3105239,3105459,3106335,3107470,3107471,3108086,3112266,3113404,3118469,3119233,3133006,3134122,3136708,3136750,3143644,3146307,3146858,3146869,3152297,3158763,3160010,3160011,3164563,3165799,3165822,3168422,3170633,3170634,3173925,3175157,3194312,3194313,3195501,3200910,3201223,3205047,3212277,3212340,3212417,3213085,3214619,3214620,3224037,3224913,3224914,3224918,3224919,3226377,3231820,3241596,3251965,3252663,3252671,3252704,3260321,3268009,3279445,3279449,3281675,3286493,3286494,3286495,3286496,3286497,3286498,3286499,3286500,3296113,3296114,3301521,3302669,3302735,3302803,3302952,3305498,3306865,3310160,3314114,3315718,3315721,3320130,3321801,3321806,3324791,3324948,3325183,3325184,3344574,3345868,3351825,3353528,3356310,3356321,3360999,3363360,3368122,3370015,3370310,3370903,3372500,3375112,3376504,3380495,3380496,3380497,3380498,3381124,3382035,3383375,3386604,3388647,3397029,3401820,3407304,3407308,3407316,3418848,3418910,3419188,3419191,3423398,3425342,3425343,3426066,3428535,3428539,3428541,3430776,3430785,3437254,3437255,3442444,3447684,3457294,3457297,3458369,3458396,3460110,3472709,3477090,3491709,3492736,3493151,3498403,3500635,3501351,3501408,3507081,3509355,3509356,3514397,3515343,3516538,3516543,3519691,3522876,3527271,3529509,3531430,3545862,3549755,3549758,3549761,3549762,3555724,3556188,3560388,3563652,3564042,3574628,3574629,3575242,3576294,3583153,3589942,3589943,3590439,3593389,3596515,3596516,3596857,3596861,3601231,3605457,3605697,3605698,3612072,3612075,3612080,3612086,3612109,3612813,3615077,3626226,3626227,3632290,3635304,3637710,3637711,3638391,3648306,3648326,3648371,3648375,3648376,3648389,3648390,3648818,3651845,3651904,3657886,3657887,3660570,3660574,3666201,3666202,3678114,3678948,3681448,3685324,3696545,3696546,3702280,3702281,3710864,3714313,3725947,3730706,3730757,3730759,3742622,3742632,3743271,3747522,3749915,3750273,3751718,3757198,3762631,3762835,3765647,3777828,3782223,3789358,3795229,3799400,3805365,3810297,3811891,3820020,3821719,3822897,3824111,3824112,3825953,3830668,3832521,3833910,3834185,3838439,3839162,3839163,3839164,3839165,3840138,3840141,3840482,3843615,3843617,3845565,3845566,3848286,3848598,3850878,3850887,3850888,3852507,3856753,3856756,3859189,3865758,3867637,3868772,3869579,3870199,3871337,3871338,3872545,3872546,3876561,3876564,3880936,3880937,3886855,3887814,3905852,3905853,3910608,3913789,3917254,3917273,3918328,3918330,3918332,3918333,3918335,3919027,3922396,3922978,3928266,3928273,3928321,3928322,3928326,3928455,3928456,3933984,3936973,3938649,3951626,3951778,3952386,3952810,3954487,3958071,3958072,3960664,3965059,3970116,3975854,3980994,3981674,3983579,3983584,3992196,3993017,3993090,3994553,3999367,4000002,4000003,4000505,4000506,4000507,4000508,4000509,4000510,4002186,4003541,4003542,4007259,4017422,4019213,4019215,4019865,4024536,4024537,4024977,4031535,4031536,4031840,4041810,4050852,4054991,4061859,4070434,4070619,4070851,4076345,4077395,4078467,4084966,4087590,4087591,4087592,4087593,4087598,4087599,4090263,4090606,4091323,4092000,4096789,4097029,4097030,4099206,4101282,4106553,4112715,4124952,4136616,4147751,4153037,4160829,4164735,4168993,4169054,4169920,4170493,4184629,4193735,4193745,4196308,4196312,4196369,4197824,4203255,4203381,4217693,4220079,4231961,4232630,4233096,4235224,4237026,4237027,4237028,4237272,4237782,4241857,4245552,4245553,4247935,4248677,4256505,4256864,4261816,4262141,4269496,4269995,4288507,4290950,4294953,4297550,4308580,4438405,4442379,4445758,4445776,4447250,4447251,4447252,4452877,4452878,4452879,4462028,4462833,4463588,4464406,4466234)

Desktop / Server (please complete the following information):

• OS & Version: Ubuntu 22.04.3 LTS
• Docker Version: 24.0.5
• Nominatim Version: 4.2.0

Additional context
docker-compose:

version: "3.7"

services:
    nominatim4:
        image: mediagis/nominatim:4.2
        environment:
            PBF_URL: https://download.geofabrik.de/europe-latest.osm.pbf
            REPLICATION_URL: https://download.geofabrik.de/europe-updates/
            NOMINATIM_PASSWORD: "${NOMINATIM_PASSWORD}"
            IMPORT_WIKIPEDIA: 'true'
            THREADS: 9
            UPDATE_MODE: "continuous"
        shm_size: '16gb'
        volumes:
            - nominatim-data:/var/lib/postgresql/14/main
            - nominatim-flatnode:/nominatim/flatnode
            - type: tmpfs
              target: /dev/shm
              tmpfs:
                  size: 16384000000 # (this means 16GB)
        networks:
            - webgateway
        deploy:
            restart_policy:
                condition: any
                delay: 0s
                max_attempts: 0
                window: 0s
            update_config:
                order: stop-first
  
networks:
  webgateway:
    driver: overlay
    external: true
volumes:
    nominatim-data:
    nominatim-flatnode: