high memory usage

[helmut72]

I run dagu with a single instance for about 9 days. Still testing dagu with 3 cronjobs that runs every few minutes. RAM usage starts at around 15MB. After 9 days it use 92MB RAM. In the past it even use over 300MB RAM with a longer uptime.

[yohamta]

Thanks for reporting the issue. Let's try to fix this.

[helmut72]

Thank you. I have started a second instance with example cronjobs. If RAM usage goes up too after some days, I'll share this example.

[helmut72]

My test instance starts at about 10 MB RAM and now need 60 MB RAM. In 3 days.

Command:

docker run -d --name dagu-test --rm -p 8080:8080 -v ./dags:/home/dagu/.dagu/dags -v ./dagu/data:/home/dagu/.dagu/data -v ./logs:/home/dagu/.dagu/logs yohamta/dagu:latest dagu start-all

Using 4 dags from your example:

./dags/hello1.yaml:

name: hello1 

schedule: "*/9 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

./dags/hello2.yaml:

name: hello2 

schedule: "*/3 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

./dags/hello3.yaml:

name: hello3 

schedule: "*/5 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

./dags/hello4.yaml:

name: hello4 

schedule: "*/7 * * * *"

steps:
  - name: get fake json data
    executor: http
    command: GET https://jsonplaceholder.typicode.com/comments
    script: |
      {
        "timeout": 10,
        "headers": {},
        "query": {
          "postId": "1"
        },
        "body": ""
      }

My other instance now runs for 12 days and need 122 MB RAM.

Btw, running of course your latest v1.12.11 release. Thank you.

[yohamta]

Thanks a lot! Will try to pprof using these examples.

[helmut72]

Can you reproduce the memory leak?

[yohamta]

Sorry, not yet because of my bandwidth, but I have removed some unnecessary heap allocations to fix other problems. So it might solve the problem to some extent.

[helmut72]

Ok, will try it after a new release.

[yohamta]

Btw, would you mind clarify the context where you're running Dagu? :) I'm wondering what makes the memory usage so important on your environment.

[helmut72]

Some Raspberry Pi provides only 1GB of RAM or even less (512MB!). It makes you sensible for RAM usage. Any swapping means higher wearing of the SD card and you really don't like a system that is crashed for this reason.

But regardless of this, I think any memory leak is something that should be fixed:
https://en.wikipedia.org/wiki/Memory_leak

[yohamta]

I see, thanks for your clarification.