Dose pixi.js can provide a async parallel loader ?

[Lyoko-Jeremie]

Hi~ everyone.
I'm a new user on pixijs.

I learned that the pixi.loaders.Loader cannot load resources when the other are loading. and if one resources are loaded, load twice will throw a exception that cannot be catched.

So, Is it able to provide a async parallel loader that can cache the parallel load task then serialize execute it?

---

follow is my simple and ugly implement on a demo project.

###the resourcesLoader class implement###

import * as pixi from "pixi.js";
import * as _ from "lodash";

class ResourcesLoaderParams {
  loaderParams: string | any | any[];
  promises: Promise<PIXI.loaders.Resource>[] = [];
  resolves: ((value?: PIXI.loaders.Resource | PromiseLike<PIXI.loaders.Resource>) => void)[] = [];
  rejects: ((reason?: any) => void)[] = [];
}

export class resourcesLoader {
  constructor(public loader: pixi.loaders.Loader) {
  }

  loaderOptions: pixi.loaders.LoaderOptions;

  waitingList: ResourcesLoaderParams[] = [];
  loadingList: ResourcesLoaderParams[] = [];

  checkExist(urls: string): boolean {
    return !_.isNil(this.loader.resources[urls]);
  }

  get(urls: string) {
    return this.loader.resources[urls];
  }

  checkAndGet(urls: string): Promise<PIXI.loaders.Resource> {
    if (this.checkExist(urls)) {
      return Promise.resolve(this.get(urls));
    }
    return Promise.reject(this.get(urls));
  }

  loadResources(urls: string): Promise<PIXI.loaders.Resource> {
    // trim existed
    if (this.checkExist(urls)) {
      return this.checkAndGet(urls);
    }
    // check is in loading or waiting, then,  merge task
    // otherwise, create new loading task
    let Li = this.waitingList.find(T => T.loaderParams == urls);
    if (_.isNil(Li)) {
      Li = this.loadingList.find(T => T.loaderParams == urls);
    }
    let thisPromise = undefined;
    if (!_.isNil(Li)) {
      thisPromise = new Promise<PIXI.loaders.Resource>((resolve, reject) => {
        Li.resolves.push(resolve);
        Li.rejects.push(reject);
      });
    } else {
      let p = new ResourcesLoaderParams();

      p.loaderParams = urls;
      thisPromise = new Promise<PIXI.loaders.Resource>((resolve, reject) => {
        p.resolves.push(resolve);
        p.rejects.push(reject);
      });
      p.promises.push(thisPromise);

      if (this.waitingList.length == 0 && this.loadingList.length == 0) {
        this.waitingList.push(p);
        this.emitLoader();

      } else {
        this.waitingList.push(p);
      }
    }

    return thisPromise;
  }

  private emitLoader() {
    if (this.waitingList.length === 0) {
      return;
    }

    let list: ResourcesLoaderParams[] = [];

    let tempList = [];
    if (_.isArray(this.waitingList[0].loaderParams)) {
      list = [this.waitingList[0]];
      for (let i = 1; i != this.waitingList.length; ++i) {
        tempList.push(this.waitingList[i]);
      }
    } else {
      // first item confident not array
      let flag = false;
      for (let i = 0; i != this.waitingList.length; ++i) {
        if (!flag) {
          if (_.isArray(this.waitingList[i].loaderParams)) {
            --i;
            flag = true;
            continue;
          }
          list.push(this.waitingList[i]);
        } else {
          tempList.push(this.waitingList[i]);
        }
      }
    }
    this.waitingList = tempList;
    this.loadingList = list;

    // trim the loaded item
    this.loadingList = this.loadingList.filter(T => {
      if (this.checkExist(T.loaderParams)) {
        T.resolves.forEach(Tr => Tr(this.get(T.loaderParams)));
        return false;
      }
      return true;
    });

    if (this.loadingList.length === 0) {
      if (this.waitingList.length !== 0) {
        this.emitLoader();
      }
      return;
    }
    let param: any;
    if (this.loadingList.length === 1) {
      param = this.loadingList[0].loaderParams;
    } else {
      param = this.loadingList.map(T => T.loaderParams);
    }
    let loadingLoader = this.loader.add(param, this.loaderOptions).load(() => {
      this.loadingList.forEach(T => {
        console.log(T.loaderParams);
        T.resolves.forEach(Tr => Tr(this.loader.resources[T.loaderParams]));
      });
      this.loadingList = [];
      this.emitLoader();
    });
    // try catch error,  example double load
    // but seemingly cannot catch it
    loadingLoader.on("error", () => {
      this.loadingList.forEach(T => {
        console.log(T.loaderParams);
        T.rejects.forEach(Tr => Tr(T.loaderParams));
      });
      this.loadingList = [];
      this.emitLoader();
    });
    loadingLoader.onError.once(() => {
      this.loadingList.forEach(T => {
        console.log(T.loaderParams);
        T.rejects.forEach(Tr => Tr(T.loaderParams));
      });
      this.loadingList = [];
      this.emitLoader();
    });

  }

}

follow is the usage:

parallel create load request and serialize load it then async call the promise "then" callback at the resource loaded immediately.
and if a resource be loaded, the "resourcesLoader" will fast resolve that request with not wait.
and if a resource are loading or waiting, the new same request will merge to the loading or waiting task.

this.loader = new resourcesLoader(pixi.loader);
this.loader.loadResources("/api/static/1.png").then(T => {
      this.image1 = new pixi.Sprite(T.texture);
    // ......
});
this.loader.loadResources("/api/static/2.png").then(T => {
      this.image2 = new pixi.Sprite(T.texture);
    // ......
});
this.loader.loadResources("/api/static/3.png").then(T => {
      this.image3 = new pixi.Sprite(T.texture);
    // ......
});

// ........

check and fast read/load resources

  updateImage(imageUrl: string) {
    this.loader.checkAndGet(imageUrl).catch(E => {
      return this.loader.loadResources(imageUrl);
    }).then(T => {
      this.image = new pixi.Sprite(T.texture);
      // .......
    });
  }

---

this simple implement on above only can load url now. but i think it can be upgrade to similar to the original loader API.

So, Can pixi.js offical be provide a async parallel loader like this?

[ivanpopelyshev]

PixiJS uses one of our core contributor pet project https://github.com/englercj/resource-loader/

Lets look in the sources:

https://github.com/englercj/resource-loader/blob/master/src/Loader.js#L20

https://github.com/englercj/resource-loader/blob/master/src/Loader.js#L446

Ok, lets look in docs, may be pixi didnt include docs for that thing because its in another module: http://pixijs.download/dev/docs/PIXI.loaders.Loader.html , here it is, concurrency.

Anyway, there are many parts of pixi that can be done better for specific project and I encourage people to use their own implementations when its possible, that suit their project. I bless you for using custom loader, please make it into a plugin and we'll reference it in plugins list.

[englercj]

I learned that the pixi.loaders.Loader cannot load resources when the other are loading

Not sure where you learned this, it absolutely can (and does) load resources in parallel. Your wrapper around resource loader just adds a second async queue on top of the one already used inside resource loader. It even has a constructor param to configure how many resources to load concurrently at a time.

if one resources are loaded, load twice will throw a exception that cannot be catched.

You can catch the error, but you don't need to. The point of that error is to tell you are using the loader incorrectly. What you should do for caching is store resources somewhere outside the loader and use a .pre() middleware to skip loading of resources that are cached. There is an example of a simple memory caching middleware in the repo and an example of using it as a .pre() middleware in the readme.

If you plan to use a loader instance multiple times you must call .reset() before using it again. That will clear the loader's state and it will be ready to load more data. It also clears the .resources object (no deletion, just drops references) so make sure you used or stored the loaded resources elsewhere before calling .reset().

[Lyoko-Jeremie]

@englercj

but if you use follow code to load resources :

    pixi.loader.add("/api/static/1.png")
      .load(() => {
        console.log("/api/static/1.png");
      });
    pixi.loader.add("/api/static/2.png")
      .load(() => {
        console.log("/api/static/2.png");
      });
    pixi.loader.add("/api/static/3.png")
      .load(() => {
        console.log("/api/static/3.png");
      });

it will case :

Error: Uncaught (in promise): Error: Cannot add resources while the loader is running.

and i get the reason from where #4100 , the loader cannot parallel load it on root resources.
so , i think it maybe can have a way to parallel create load task on root resources.

anyway, thanks your contribute.

[Lyoko-Jeremie]

@ivanpopelyshev thank you~
I'm glad to make a plugin, and need some times to learn how to do it.
so, where can find the documents about How To Make A PIXI Plugin ?

[ivanpopelyshev]

There are docs about renderer plugins, but for everything else the rule is "do whatever you want, give users JS file that to be included after vanilla pixi.js file".

Pixi is built with classes, there are no hidden variables in hidden contexts.

[Lyoko-Jeremie]

@ivanpopelyshev thank you~

[themoonrat]

@Lyoko-Jeremie you are using the loader incorrectly. The correct usage is to add all of the resources you want to load, then you call the load() function. Currently, you are calling load after every add.

[Lyoko-Jeremie]

@themoonrat but in my case, which resources need load cannot know before it need to load.
I'm writing a demo like maps . in this case, load which pice image are depend on user who need see where.
user maybe move faster than the load speed that will case the issue Cannot add resources while the loader is running. .
and i cannot pre load all the image to browser because all the image is so large.

[ivanpopelyshev]

@Lyoko-Jeremie You can abuse dependant resource mechanism.

Make a resource that never actually loads (middleware runs forever), add children to it.

Spine loader waits for two extra child resources to load: https://github.com/pixijs/pixi-spine/blob/master/src/loaders.ts#L7 , you can make a middleware that waits forever.

[ivanpopelyshev]

As another option, you can take resource and queue from resource-loader, but make your own Loader class that stores resources differently.

[Lyoko-Jeremie]

@ivanpopelyshev how to make a resource that never acutally loads ?
I don't find this info from other place, i think this warnning need writing to guid to let new user can know that. because i think it's a anti-conscious feature.

[ivanpopelyshev]

@Lyoko-Jeremie

If you need a custom loader for your game - you have to code it yourself. Its better if you salvage parts that are ok or hack existing code to save the time, but for that you have to learn all the code from 1) resource-loader repo 2) all pixi middlewares 3) advanced middlewares (Spine).

I can answer your questions after you spend some hours on studying all that code.

Alternatively, you can look how fromImage works, its easier, it uses cache and you might do something like that.

[Lyoko-Jeremie]

@ivanpopelyshev I like to do that, but maybe no time give me to do it.
in now day, this wrapper resourcesLoader are enough to this demo.
and i will try to read the loader code on my free time.
thank you very much.

[themoonrat]

As an idea for others having this issue, one option could be to have a pool of resource loaders. If you need a resource loading, but existing loaders are already busy, create a new loader and load the resource that way. If an existing loader had finished, reset it and reuse. Then build your own wrapper class to have a single 'load this asset' function, and it manages the resource loaders in the background

[Lyoko-Jeremie]

@themoonrat Good idea !!!
so, can create any more resource loader without any other limit ? and don't need bind to any pixi's compoment ?

[themoonrat]

Yeps
If you want to use the PIXI version with it's own middleware, then you'd use

const loader = new PIXI.loaders.Loader();

Loaders are very lightweight, so don't worry about having multiple of them.

The only thing I'd be aware of, is that the loaders let you set how many assets it can concurrently download, which has a default of 10. If you had 3 loaders, you could be effectively trying to load up to 30 resources at once (tho I'm sure browser would limit lower than that). So maybe you'd create each loader with a lower limit? Or maybe your primary loader has a large limit to get the initial assets down, but your 'in-game-streaming- asset loaders have a lower limit to focus on getting each individual asset down faster.

[Lyoko-Jeremie]

@themoonrat thank you for your explain~~ I love you~

[bigtimebuddy]

This seems answered, thank you all for your response.

[Lyoko-Jeremie]

I think on the first, I will change my resources loader to pool version, and each loader load one request. This don't need many times.

But I still have a question, how to catch the error of Resource with name "…" already exists ?

[ivanpopelyshev]

Check that resource with name "..." already exists in the same loader.resources before you actually add it.

Again, with the nature of your project (BIG MAP), I advice you to change the attitude, meditate for some time and prepare for hundreds of questions like that.

It might be that in a few days time you'll start forum search about "how to show big map" and stumble across my countless posts about pixi-tilemap, graphics, meshes, e.t.c.

UPD. we are ready to share experience and answer on very difficult questions

[Lyoko-Jeremie]

oh~ good forum. thank you~

[Lyoko-Jeremie]

i think my case maybe have some different from a game.
all the pice is real world satellite like image , have big size (20MB per pice), and irregular shape with many translucent , hole , rotate and overlapping parts.
You can imagine that this is a fake google maps, with non-Tiles image pice, because in some reason it cannot stitching and segmentation in server side.

so, will have some different question on this, but i will try to find solutions with very hard work. (not knowing whether to cry or to laugh..)

[ivanpopelyshev]

not knowing whether to cry or to laugh..

Yes! That's the spirit!

all the pice is real world satellite like image , have big size (20MB per pice),

Is the target mobile or PC? For PC you may use compressed textures (dds + static gzip on server + pixi-compressed-textures).

On mobile you'll have serious lag when pixi uploads that into gpu, we dont have progressive uploader, yet. There are issues about it.

[ivanpopelyshev]

Basically, you need a view 2x of the camera, and when camera touches the edge of "prepared" rectangle you add more objects, and put the old ones into some kind of queue. Right now pixi gc unloads from videomemory any texture that doesnt get rendered in 4 minutes or so. (PIXI.settings.GC_MODE, renderer.texture_gc.mode). You need your own queue anyway because you store loaders and initial downloaded data within them, to be re-used if user moves camera back.

[Lyoko-Jeremie]

thanks for your notice~

now it's not on mobile , but who know the Unpredictable future ?

in other side. By some reason, many future's user maybe use the older broswer that dont support WebGL. (this reason you may not know if you never develop a program for Chinses people.)
so, i also need to test the canvas' performence when demo is OK, and make the decision at that time that , whether or not to show a Banned page to older brwser users and let they to install a new browser. (Helplessly and stall hands...)

[Lyoko-Jeremie]

and I have a Unfragmented Zoom feature, this means Unlimited Zoom. (Doge Face~)

[ivanpopelyshev]

If you are sure about your target audience , do it for canvas and test it for canvas. If its possible to actually measure audience, if you know your traffic source, you can estimate how many people have webgl enabled in their browser. If its 99.5%, use webgl-only. IF its win-xp and IE10 then.. well... good luck :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.