eigen and linalg

Some facts

• linalg api has saturated and doesn't seem to change a lot these days. This means we mostly need the types of calls that are in there: a number of dot products, linear solves, factorizations
• Compile time and memory of linalg modules are now a problem
• Maintenance is a problem as the namespace/macro files are very big (changes are costly in labor)
• GPU features of linalg are not used in algorithms
• eigen3 is the only backend
• autodiff won't work with the current system
• due to the lack of expression trees at compile or runtime, no heterogeneous computing devices can be exploited

pro eigen

• simplicity, less code, outsourcing
• linalg has many issues even though although we tried to write something that lasts. Maybe we shouldn't do this ourselves? This in particular includes GPU/CPU/mix stuff
• eigen3 is stable here to stay, e.g. as it is a dependency for tensorflow, back then eigen3 was way more niche than now
• eigen's dsl is more clean than our linalg API, which is quite cumbersome as not oop, epecially when doing non-trivial dot product chaining (need to pass transpose flags as function arguments)
• A solution for GPU/CPU/etc will probably be built on top of eigen (by somebody else), see e.g. sycl
• We like compilers, shogun modules are usually fixed, so why not do compile time optimized linear algebra
• Eigen has minimal autodiff support built in (for scalars), see Gil's patch. We could probably extend this to vector valued expressions

con eigen

• We had eigen to replace lapack, then we built linalg. The point was not to be dependent on a single lib, but have something that can be easily interchanged (at runtime even). There is a danger that tomorrow a new lib comes about and we have to refactor everything again.
  • The question is whether this is feasible, as we would have to make assumptions on how a future lib will work, and anticipate designs ... very difficult
  • An example of how difficult this anticipation is, is our the GPU stuff...which we never even made work.
  • With unlimited manpower, we could built the thing we want. But we don't have that.
• compile time increases
  • plugins would at least allow to compile only algorithms that are wanted

The optimal solution

• Expression trees built at runtime that are JIT'ed

  • allows for easy autodiff, one can have code to build expressions and then access this from different parts of the framework
  • should be as fast as compiled (?)
  • writing this ourselves is nuts, do frameworks for this exist?
  • need to refactor all algos

• Expression trees built are built at compile time and we rely on compiler to optimize/distribute

  • should allow for autodiff, although not as elegant from a framework perspective (re-using expressions tricky) (that's what eigen's autodiff does)
  • still fast
  • fits shogun more as our models are fixed at compile time -> less heavy refactoring if any