Efficiency issues with fpm for large dependencies

[zoziha]

When I use fpm for coding, building the dependency model and compiling the code becomes visibly slow when the dependencies are large.

Especially under Windows system, it seems to be much slower to access the file system than Linux and macOS.
The current strategy of fpm is to first retrieve all source files, build a dependency model, and then start compiling. On Windows systems, for larger code, it will take several seconds to build a dependency model in every fpm build, which will give users a feeling that fpm is very slow.

The following issues are relevant to this discussion and may improve this situation:

• Only compile source of dependencies needed for project #635
• optimize file listing #507

• fpm/src/fpm.f90

  Line 157 in 95dbca1

  do i = 1, model%deps%ndep

I don't learn Rust and I don't understand how cargo handles this: in particular, there are enough dependent libraries and the total amount of code is large.

Is there a good way to improve this problem:

• Enhance file system access in fpm under Windows?
• Improve the build dependency model, compilation process? Asynchronous, parallel?
• Use cache to store dependency model?
• Distinguish the impact of local code and fpm dependencies on compilation. For example, dependencies are often not changed frequently, while local project code is frequently changed?

[LKedward]

Thanks @zoziha and apologies for the late response! I agree this is an area where fpm could do with some improvement. It may be possible to parallelize some parts of the startup processing but it's possible that the bottleneck here is actually disk io when parsing files. Similarly, there are definitely areas where the algorithms could be improved and optimized - again this will require profiling to identify the hostpots.

I like your suggestion to cache model information for dependencies and IMO that will probably provide the greatest improvement for the fewest (complex) code changes. It should be quite straightforward to cache model information for dependencies to the build folder and clear the cache if the dependency gets updated.

---

Out of interest, which method do you use build/install fpm on Windows?

[zoziha]

Thank you @LKedward very much for your reply, I think that using fpm for io on windows will be much slower than linux/macOS. I originally planned to use windows-mingw64-gprof to analyze the efficiency bottlenecks when building projects with fpm. Unfortunately, I need to find a way to use a right windows-gprof version to do this (see why-gprof-gives-empty-log-file-on-windows-10).

Out of interest, which method do you use build/install fpm on Windows?

Because I pay more attention to the update of the fortran-lang organization, on a new computer, I usually download the .exe file in the release first, then pull the fpm source code, and use path/to/fpm install --profile release --flag "-fopenmp -march=native -flto -g -fbacktrace" (I may think that In this way, the efficiency of fpm is the best) to install fpm.

Unfortunately, the speed of accessing github from China is often not good, which also prevents more fortran Chinese users from accessing and installing fpm :)

[zoziha]

I got the gprof report running on Windows (use Windows-Octave GFortran), the gprof report only has information about function calls, they run very fast, there is no stats for disk IO, I also tried to use time to get the running time of fpm to build the dependency tree (time fpm build --show-model), can infer, for large projects, a large number of disk reads and writes really slow down fpm's building of the dependency tree, and the CPU is hovering and waiting during this process. (In fact, disk IO is indeed much slower than CPU processing.)

I feel like we can use the disk cache to store the dependency tree (maybe a TOML table), and the second time we run fpm, fpm read the dependency tree cache and only reprocess the modified source files. (For short term, I like this idea)
Or use the method recommended by @awvwgk to dynamically generate a dependency tree during the build process, which may make more changes to the fpm code. (For long term, I like this idea)

[zoziha]

Like the ability to use openmp to build code in parallel (very efficient), it seems that parallelism can also be performed in file IO parsing based on source files in the same folder (fpm_sources.::add_sources_from_dir), but this requires refactoring file parsing errors.

fpm/src/fpm_sources.f90

Line 96 in 87dcb9b

if (allocated(error)) return

It seems possible to set up a parallel domain at Line93 to Line112 in fpm_sources.f90.

[zoziha]

I tried to parallelize the source file parsing (see parallel-readfile for details, don't know if I did it right), but the end result didn't improve.

It may be due to the Windows NTFS file system or the MingW environment (impure/non-Windows native environment). It still takes some time for fpm to parse the source files of large projects under Windows, and it seems that the problem cannot be solved. There may be two ways to solve the problem:

1. While parsing the source file, build the dependency tree and compile it at the same time;
2. Try to compile fpm with native Fortran compiler under Windows, such as intel ifort/icc.

(While fpm under Linux system, parsing files is almost instantaneous ~)

[LKedward]

Thanks for the update @zoziha and for trying that out. Yes, I had suspected that parallelization might not help us here due to an IO bottleneck.

Suggestion 1. is quite complex and I myself am not sure of the best way of implementing it for efficiency.
Suggestion 2. is intriguing and I'm interested to see whether compiling fpm with ifort does alleviate these issues at the parsing stage.

My personal opinion is that caching the model information for dependencies is the way to go since it can be easily incorporated into the existing code structure and would bypass the need to read and parse those source files repeatedly.

[ivan-pi]

Perhaps you can time the execution of ifort -gen-dep=dep.dd -syntax-only src/*.f90, which should give you a dependency graph in Makefile format. This would give us an estimate of how long the compiler parser needs and how much better/worse fpm is. Ideally fpm would need less time, since it doesn't inspect syntax. Repeat without -syntax-only to get an idea of the ratio between parsing vs parsing and compiling.

A somewhat crazy idea would be to read the file as an integer array and then sweep through it using integer masks potentially exploiting SIMD instructions, for inspiration see:

• Accelerating intersections with SIMD instructions
• How quickly can you remove spaces from a string?
• Removing duplicates from lists quickly
• Parsing Gigabytes of JSON per Second
• Parsing JSON Really Quickly: Lessons Learned

[zoziha]

It seems that it is more promising to store model dependency information, which may depend on #541. When we have model information stored in TOML file, it will be possible to use it to achieve subsequent evolution.

At the same time, I noticed an interesting read file implementation read_whole_file in toml-f, which reads the entire text file as a string at once, which may be a good way to improve reading efficiency, perceptually see, reading as a string in one go should be faster than reading line by line.