Performance problem on require, require_once, include php statements

[MarcelloOr]

Hi all,
I'm experimenting bad performance and the problem seems to be when calling require, require_once, include statements.

Here's a very simple file test.php I used to test the problem. The file simply load 3 files test1.php, test2.php and test3.php.

<?php
$start = microtime(true);
require('./test1.php');
$stop = microtime(true);
echo "require: " . ($stop - $start) . '<br>';

$start = microtime(true);
require_once('./test2.php');
$stop = microtime(true);
echo "require_once: " . ($stop - $start) . '<br>';

$start = microtime(true);
include('./test3.php');
$stop = microtime(true);
echo "include: " . ($stop - $start) . '<br>';

Files test1.php, test2.php and test3.php are as follow:

<?php
echo 1; // 2 and 3 in the corrisponding file

When I call test.php from the container (using the image php:7.0-apache), the response time is 10x worst than when I execute the same simple code directly from the host (same apache and same php7).

I'm using Debian 8 also for the Host and Docker Cloud for orchestration.
Here's the output of "docker info":
Containers: 51
Running: 16
Paused: 0
Stopped: 35
Images: 388
Server Version: 1.11.2-cs5
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 839
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge null host weavemesh
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.789 GiB
Name: xxxxxxx
ID: 2PC2:2IGQ:KT26:O4KM:RFID:HUWW:Y2FO:LYSK:MNZR:KXKY:OPFO:G2CX
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username: xxxxxxxxxx
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

[tianon]

Interesting -- are these files baked into the image (and thus part of the AUFS layering, and subject to AUFS performance characteristics on top of the host filesystem), or are they part of a volume/bind mount (and thus subject to the host filesystem performance characteristics only)?

[MarcelloOr]

@tianon I tried with 1) a volume/bind mount and 2) with files created on the fly inside the container after I started it without a volume. I had the same differences between the host and container results in both cases.
I didn't try with a new image with these files stored inside.

I also installed a fresh new debian 8 on a PC and replicated the test on the host and in the container. Same results as on the precedent host.

By the way, if I compare performance for other kind of statements (i.e. using memory), the results are instead quite the same in the host and the container. For example:

<?php

$start = microtime(true);

for($i=0; $i < 1000000; $i++){
        $j[] = $i;
}

$stop = microtime(true);
echo "require: " . ($stop - $start) . '<br>';

This point has been also verified comparing results between the host and the container of this php benchmark.

[MarcelloOr]

Update: I tried also with a very simple image containing the test files above but I have the same results on a fresh debian8 installed on a PC (performance results in the container are 10x worst than on the host).

Here's the Dockerfile I used:

FROM php:7.0-apache
COPY test.php /var/www/html/
COPY test1.php /var/www/html/
COPY test2.php /var/www/html/
COPY test3.php /var/www/html/

I don't have any other ideas. Are people correctly running PHP web applications with these PHP statements inside containers without performance problems? Are they using Debian8 as the host?

[MarcelloOr]

Hi all,
I also tested in a new fresh CentOS installation on the same PC used before for the fresh Debian 8.
Same results... container response time 10x time worst than the host.

Here's the output of "docker info" in this case:

[root@localhost html]# docker info
Containers: 12
 Running: 10
 Paused: 0
 Stopped: 2
Images: 11
Server Version: 1.11.2-cs5
Storage Driver: overlay
 Backing Filesystem: xfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: weavemesh null host bridge
Kernel Version: 3.10.0-693.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.542 GiB
Name: localhost.localdomain
ID: 2JQJ:4AES:76JH:JKBR:YVIC:EBUH:UYAN:DKF7:WF2I:4QH2:BZ56:FJKS
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/

Please note that in this case the storage driver is different from that used with Debian 8 (overlay instead of aufs).
At the end, the problem seems NOT related to:

• OS version used for the host (same problem on Debian 8 and CentOS 7)
• mount volume vs image files vs files created on the fly inside the container
• storage driver

It's seems to me, but it sounds crazy, that these kind of PHP statements (require, require_once, include) does not work correctly in docker !!!

[alexmatsak]

+1

[bweston92]

@alexmatsak can you provide docker info too.

[alexmatsak]

Containers: 39
 Running: 26
 Paused: 0
 Stopped: 13
Images: 277
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host ipvlan macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.9.49-moby
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.786GiB
Name: moby
ID: YDAN:WISP:BGZE:GZV2:CA6S:QRIH:ENOH:3QDF:IQAR:IH6B:NTNL:WDA2
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 265
 Goroutines: 330
 System Time: 2017-11-16T14:10:22.2311708Z
 EventsListeners: 2
No Proxy: *.local, 169.254/16
Registry: https://index.docker.io/v1/
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[kducharm]

This is very interesting, I wrote a PHP include test tool based on your tests. I'm not entirely sure this is anything to do with the PHP image itself - I believe it's still an issue of Docker volume mapping performance and the use of file streams.

Here is the test tool: https://github.com/kducharm/phpincludetest - it includes a flag to enable memory test in addition to just the file include tests.

I have Docker For Mac Edge (17.09.0-ce-mac34 19605), as well as VirtualBox running Ubuntu 16.04. With Docker For Mac, the volume mapped test is at least 10x slower than host, and building an image without volume mapping is roughly the same speed as the host. This doesn't surprise me given the performance issues plaguing Docker For Mac volumes, but is pretty bad.

In Ubuntu under VirtualBox on Mac, running docker (inception...) - host/volume mapped/image all perform roughly the same as host/image on Docker For Mac. I haven't tested Ubuntu enough to know what to expect, but it looks pretty performant.

I'll probably share this testing tool on docker forums, and would be good to augment it with other file I/O tests. This is only a single container test, and I've been experiencing bad performance with DB containers on Mac and Ubuntu and need to narrow it down to file I/O or some other multip[le container/volume mapping issue.

Docker machine w/NFS runs way faster on Mac than Docker For Mac currently for me, which is not good.

[lsascha]

Anyone already tried to use cached volumes if it is any faster?
See https://docs.docker.com/docker-for-mac/osxfs-caching/

[oxygen]

Using HGFS or CIFS shares, the problem is exacerbated without limits proportionally with the number of use aliases in a large application. The best I could determine was that in some scenarios there are some sort of loops even if a require_once was already executed. In my case, the warm-up time has exceeded is >=3 seconds (sometimes a lot worse) for a 100 - 200 ms warm-up without a mount.
It think it has to do with opcode caching giving up at some point, probably because file stats checking is taking too long.

My code also has lots of traits (like a huge lot of them), and it seems it might be related.

Just saying this because require performance over mounted file systems in PHP might not be Docker related.

Does opcache.revalidate_freq = 60 fix your problem after a few runs?

[zsimple]

I also found it really slow on php-fpm docker image, but if use the ubuntu image, apt install the php-fpm and nginx, everything works well... I dont know why

php7.2-fpm-alpine

  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.59s   366.04ms   1.87s    88.50%
    Req/Sec    29.24     13.61    70.00     70.31%
  574 requests in 10.01s, 1.37MB read
Requests/sec:     57.32
Transfer/sec:    139.60KB

pure ubuntu image and install php manully...

  2 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   171.79ms   38.81ms 458.92ms   73.56%
    Req/Sec   289.15     70.54   424.00     65.00%
  5762 requests in 10.01s, 13.57MB read
Requests/sec:    575.65
Transfer/sec:      1.36MB

[brndusic]

Hi All,
here you can see a difference https://medium.com/@uv_d/migrating-our-php-applications-to-docker-without-sacrificing-performance-1a69d81dcafb

I used their approach and found performance improvements but can't understand what is causing this problem.

[lsascha]

thats interesting. And thanks for sharing the story. I think i will try that solution.

The most obvious thing that is different is that they use precompiled binaries from the distribution instead of having it compiled in the docker-image like the official php image does.

One other thing i noticed,
when i used the PHP+Apache single container image, i constantly got empty responses randomly.
I think especially when the container wrote files (for caching) and after that immediately read some file (wich did not had to be one of the newly written files)
After switching to PHP and Apache containers seperately, these empty responses vanished completely.

Not exactly why i had these empty responses. When i tried the Neos CMS https://www.neos.io/
It was basicly unusable because the whole backend barely loaded completely.

[tianon]

https://medium.com/@uv_d/migrating-our-php-applications-to-docker-without-sacrificing-performance-1a69d81dcafb is way interesting, thanks for the pointer. I think we need to dig more into what's going into those Debian/Ubuntu packages and figure out why our performance is so much worse, because from what I can tell, we're already doing pretty similar things so I'd love to get to the bottom of what's going on there (and the testing there appears to be pretty systematic, which is what I've been looking for to dig more into this issue).

[axot]

I did not test this case, but we are using opcache.enable_file_override=1 for our production which will reduce proc_cred's overhead.

[mtibben]

I also was noticing performance issues vs debian, and installing opcache fixed the issues. I think opcache should be installed by default on these images