Drill Dynamic Allocation

[LYCJeff]

Does drill have dynamic allocation features similar to spark?

Or can we implement a similar function of drill on yarn or k8s based on metrics?

[jnturton]

It's not hard to do a basic form of autoscaling in k8s: https://github.com/Agirish/drill-helm-charts#autoscaling-drill-clusters

[paul-rogers]

What is your use case?

Drill differs from Spark. Spark allocates a set of workers per job. Spark starts workers for each job, based on the estimated complexity of the job. By contrast, Drill uses a shared cluster: queries run using the workers available at the moment the query runs. Spark is designed for large, complex, long-running jobs. Drill is designed for many concurrent short-running queries. In Drill, the query would normally be done long before new nodes get organized and join the cluster.

Back in the day, Drill provided Drill-on-YARN to manage a Drill cluster in Hadoop. Scaling was manual, though an API encouraged someone to design a controller that would observe load and scale the cluster up or down to track average load. In modern times, K8s is the preferred alternative. The Drill operator handles the mechanics of scale-up or -down. Again, a separate controller would be needed to adjust cluster size based on load trends and/or local policies. Feedback Control for Computer Systems explains the the kind of PID controller that could do the job.

[LYCJeff]

@jnturton @paul-rogers Thanks for reply.

In my case, queries are irregular, with a high of a few hundred concurrent and a low of completely idle. I was wondering if Drill could run serverless or elastic on the idle resources of the cluster, rather than taking up a lot of resources up front. Something like spark thriftserver with dynamic allocation.

I tried to deploy drill on k8s with HPA. As @paul-rogers said, in some cases, the query would normally be done long before new nodes get organized and join the cluster.