wand.exceptions.CacheError: cache resources exhausted `' @ error/cache.c/OpenPixelCache/4083 #544

DanKrt82 · 2021-07-16T06:33:21Z

DanKrt82
Jul 16, 2021

Hi all,

I have a problem during saving a big file from a numpy array. I use TemporaryFile, but on the line "with wandImage..." the script raise an exception like:

File "/home/prj/env/lib/python3.8/site-packages/wand/image.py", line 9106, in init
self.read(file=file)
File "/home/prj/env/lib/python3.8/site-packages/wand/image.py", line 9705, in read
self.raise_exception()
File "/home/prj/env/lib/python3.8/site-packages/wand/resource.py", line 222, in raise_exception
raise e
wand.exceptions.CacheError: cache resources exhausted `' @ error/cache.c/OpenPixelCache/4083

I try to scale up the values on /etc/ImageMagick-6/policy.xml but I have the same error, any idea about a solution of that problem?

The code:

with TemporaryFile() as tfd:
            np_array.tofile(tfd)
            tfd.seek(0)
            with wandImage(file=tfd, depth=8, format='R', width=x * 2, height=h_swath) as img:
                img.type = 'grayscale'
                img.depth = 2
                img.resolution = (int(dpiX), int(dpiY))
                img.compression = 'undefined'
                img.save(filename=out_path + str(swath) + '.tif')

The package that I use are:

$ dpkg -l | grep wand
ii  libmagickwand-6-headers              8:6.9.10.23+dfsg-2.1ubuntu11.4        all          image manipulation library - headers files
ii  libmagickwand-6.q16-6:amd64          8:6.9.10.23+dfsg-2.1ubuntu11.4        amd64        image manipulation library -- quantum depth Q16
ii  libmagickwand-6.q16-dev:amd64        8:6.9.10.23+dfsg-2.1ubuntu11.4        amd64        image manipulation library - development files (Q16)
ii  libmagickwand-dev                    8:6.9.10.23+dfsg-2.1ubuntu11.4        all          image manipulation library -- dummy package

Here my policy.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE policymap [
  <!ELEMENT policymap (policy)+>
  <!ATTLIST policymap xmlns CDATA #FIXED ''>
  <!ELEMENT policy EMPTY>
  <!ATTLIST policy xmlns CDATA #FIXED '' domain NMTOKEN #REQUIRED
    name NMTOKEN #IMPLIED pattern CDATA #IMPLIED rights NMTOKEN #IMPLIED
    stealth NMTOKEN #IMPLIED value CDATA #IMPLIED>
]>
<!--
  Configure ImageMagick policies.

  Domains include system, delegate, coder, filter, path, or resource.

  Rights include none, read, write, execute and all.  Use | to combine them,
  for example: "read | write" to permit read from, or write to, a path.

  Use a glob expression as a pattern.

  Suppose we do not want users to process MPEG video images:

    <policy domain="delegate" rights="none" pattern="mpeg:decode" />

  Here we do not want users reading images from HTTP:

    <policy domain="coder" rights="none" pattern="HTTP" />

  The /repository file system is restricted to read only.  We use a glob
  expression to match all paths that start with /repository:

    <policy domain="path" rights="read" pattern="/repository/*" />

  Lets prevent users from executing any image filters:

    <policy domain="filter" rights="none" pattern="*" />

  Any large image is cached to disk rather than memory:

    <policy domain="resource" name="area" value="1GP"/>

  Define arguments for the memory, map, area, width, height and disk resources
  with SI prefixes (.e.g 100MB).  In addition, resource policies are maximums
  for each instance of ImageMagick (e.g. policy memory limit 1GB, -limit 2GB
  exceeds policy maximum so memory limit is 1GB).

  Rules are processed in order.  Here we want to restrict ImageMagick to only
  read or write a small subset of proven web-safe image types:

    <policy domain="delegate" rights="none" pattern="*" />
    <policy domain="filter" rights="none" pattern="*" />
    <policy domain="coder" rights="none" pattern="*" />
    <policy domain="coder" rights="read|write" pattern="{GIF,JPEG,PNG,WEBP}" />
-->
<policymap>
  <policy domain="resource" name="memory" value="10GiB"/>
  <policy domain="resource" name="map" value="10GiB"/>
  <policy domain="resource" name="width" value="200KP"/>
  <policy domain="resource" name="height" value="200KP"/>
  <policy domain="resource" name="area" value="1GiB"/>
  <policy domain="resource" name="disk" value="5GiB"/>
  <policy domain="delegate" rights="none" pattern="URL" />
  <policy domain="delegate" rights="none" pattern="HTTPS" />
  <policy domain="delegate" rights="none" pattern="HTTP" />
  <policy domain="path" rights="none" pattern="@*"/>
  <policy domain="coder" rights="none" pattern="PS" />
  <policy domain="coder" rights="none" pattern="PS2" />
  <policy domain="coder" rights="none" pattern="PS3" />
  <policy domain="coder" rights="none" pattern="EPS" />
  <policy domain="coder" rights="none" pattern="PDF" />
  <policy domain="coder" rights="none" pattern="XPS" />
</policymap>
<!--<policymap> -->
  <!-- <policy domain="system" name="shred" value="2"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- <policy domain="system" name="memory-map" value="anonymous"/> -->
  <!-- <policy domain="system" name="max-memory-request" value="256MiB"/> -->
  <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
  <!-- <policy domain="resource" name="memory" value="256MiB"/> -->
  <!-- <policy domain="resource" name="map" value="512MiB"/> -->
  <!-- <policy domain="resource" name="width" value="16KP"/> -->
  <!-- <policy domain="resource" name="height" value="16KP"/> -->
  <!-- <policy domain="resource" name="list-length" value="128"/> -->
  <!-- <policy domain="resource" name="area" value="128MB"/> -->
  <!-- <policy domain="resource" name="disk" value="1GiB"/> -->
  <!-- <policy domain="resource" name="file" value="768"/> -->
  <!-- <policy domain="resource" name="thread" value="4"/> -->
  <!-- <policy domain="resource" name="throttle" value="0"/> -->
  <!-- <policy domain="resource" name="time" value="3600"/> -->
  <!-- <policy domain="coder" rights="none" pattern="MVG" /> -->
  <!-- <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" /> -->
  <!-- <policy domain="delegate" rights="none" pattern="HTTPS" /> -->
  <!-- <policy domain="path" rights="none" pattern="@*" /> -->
  <!-- <policy domain="cache" name="memory-map" value="anonymous"/> -->
  <!-- <policy domain="cache" name="synchronize" value="True"/> -->
  <!-- <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/> -->
  <!-- <policy domain="system" name="pixel-cache-memory" value="anonymous"/> -->
  <!-- <policy domain="system" name="shred" value="2"/> -->
  <!-- <policy domain="system" name="precision" value="6"/> -->
  <!-- not needed due to the need to use explicitly by mvg: -->
  <!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
  <!-- use curl -->
  <!-- <policy domain="delegate" rights="none" pattern="URL" /> -->
  <!-- <policy domain="delegate" rights="none" pattern="HTTPS" /> -->
  <!-- <policy domain="delegate" rights="none" pattern="HTTP" /> -->
  <!-- in order to avoid to get image with password text -->
  <!-- <policy domain="path" rights="none" pattern="@*"/> -->
  <!-- disable ghostscript format types -->
  <!-- <policy domain="coder" rights="none" pattern="PS" /> -->
  <!-- <policy domain="coder" rights="none" pattern="PS2" /> -->
  <!-- <policy domain="coder" rights="none" pattern="PS3" /> -->
  <!-- <policy domain="coder" rights="none" pattern="EPS" /> -->
  <!-- <policy domain="coder" rights="none" pattern="PDF" /> -->
  <!-- <policy domain="coder" rights="none" pattern="XPS" /> -->

<!-- </policymap> -->

Thank you,

Daniel

DanKrt82 · 2021-07-16T06:35:18Z

DanKrt82
Jul 16, 2021
Author

The package that I use are:

$ dpkg -l | grep wand
ii  libmagickwand-6-headers              8:6.9.10.23+dfsg-2.1ubuntu11.4        all          image manipulation library - headers files
ii  libmagickwand-6.q16-6:amd64          8:6.9.10.23+dfsg-2.1ubuntu11.4        amd64        image manipulation library -- quantum depth Q16
ii  libmagickwand-6.q16-dev:amd64        8:6.9.10.23+dfsg-2.1ubuntu11.4        amd64        image manipulation library - development files (Q16)
ii  libmagickwand-dev                    8:6.9.10.23+dfsg-2.1ubuntu11.4        all          image manipulation library -- dummy package

4 replies

emcconville Jul 16, 2021
Maintainer

Please update the original question w/ the additional details. Also include the policy.xml

DanKrt82 Jul 16, 2021
Author

Updated, thank you.

emcconville Jul 16, 2021
Maintainer

Cool. I'm guessing the policy gave ImageMagick more resources then can be used by a single process. Remember you still have large numpy arrays in memory. Try dropping the area policy down to the default 126MiB value. I can't remember if area, or map, properties control cache-page size -- so you might want to try dropping both values. To fix this issue, we want ImageMagick's resource manager to use disk cache.

DanKrt82 Jul 16, 2021
Author

Hi,

thank you for the reply.

I try to dropping down to the default value the area policy but I have the same error, also dropping the map value, but the error still appear. Can I try with some other parameters?

Thank you

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wand.exceptions.CacheError: cache resources exhausted `' @ error/cache.c/OpenPixelCache/4083 #544

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

wand.exceptions.CacheError: cache resources exhausted `' @ error/cache.c/OpenPixelCache/4083 #544

DanKrt82 Jul 16, 2021

Replies: 1 comment · 4 replies

DanKrt82 Jul 16, 2021 Author

emcconville Jul 16, 2021 Maintainer

DanKrt82 Jul 16, 2021 Author

emcconville Jul 16, 2021 Maintainer

DanKrt82 Jul 16, 2021 Author

DanKrt82
Jul 16, 2021

Replies: 1 comment 4 replies

DanKrt82
Jul 16, 2021
Author

emcconville Jul 16, 2021
Maintainer

DanKrt82 Jul 16, 2021
Author

emcconville Jul 16, 2021
Maintainer

DanKrt82 Jul 16, 2021
Author