Resource update architecture

global web disco archi

Bookmarklet :

• v1 Phase in UI (display & choose) & discovery.js sendWSDL() (lookup "TODO v1 Phase")
• p1,5 list environments but still allow to write a non existing one

dbb (any) :

• v1 Phase & env in UI (display only) & browser + node js (lookup "TODO v1 Phase")
• LATER unify bookmarklet & proxy dbb at least on the side of node js, if possible on the side of browser js (jsonp vs json methods...) ; put jsonp version rather on the node side...

MatchingDashboard (see below) :

• LATER allow to create service from endpoint
• LATER allow to set "dontMatch" so it can be unmatched without auto rematch until explicit matching

OPT unify matching with discovery i.e. allow to create new SOA iserv & impl or match existing ones at will :

• NB. by default (monit &) web disco creates endpoints (TODO OR WebResources ?!?) that are exactly identified by their subproject + env + url, so this is about an additional step
• by completing disco'd info to make sure it matches existing SOA nodes or not (but matches platform, component...)
• this info may be : 1. provided in MatchingDashboard, 2. conf'd in disco session, 3. for each WSDL, 4. OPT refined in UI after first tentative matches
• 1. provided in MatchingDashboard :
• only endpoint is created,
• (if possible) wether it has (been matched to) impl & iserv is displayed using their icons & links
• a (at worse global) link to the MatchingDashboard is provided
• 2. disco session conf :
• in url, at the beginning : subproject, env, (dontCreateIfNoMatch), OPT component (discover app, workflow...) & platform (i.e. create component)
• 2. OPT per WSDL conf :
• BY DEFAULT dontCreateInformationService (dontCreateServiceImplementation) if no single match
• TODO dontMatch by setting a fake "toBeManuallyMatchedComponent" component (or prop), TODO enrich MatchingDashboard to single it out as "matching inbox"
• select component & / or platform (or create one, same as filling the disco'd node's platform info). Like MatchingDashboard.fetchComponents/Platforms() TODO still
• TODO HOWEVER maybe best to reuse MatchingDashboard rather than copy / paste, all the more since it has to be completed with matching anyway
• 3. OPT refine :
• for each found WSDL, [if endpoint not found TODO] find matching impls or else iservs, then allow to force creation or specify guided or exact match
• how : like MatchingDashboard.fetchSuggestions() but remotely (WebEngine) & using temp endpoint or else impl documentModel (like in DiscoveryService matchFirst), then MatchingDashboard's component & platform refining
• alt 1 : TODO adapt MatchingDashboard to for temp documentModel : GET takes an optional "iservs" query param, dbb ui ?!? (to adjust component & platform...), dbb builds SoaNodeInformation json model BUT TODO PB misses WSDL extracted info
• alt 2 : TODO create a disco session-specific subproject, from which it is later copied in actual subproject HOWEVER TODO not OK for matching existing service YES IT IS BY CREATING FAKE ENDPOINT THAT IS NOT COPIED AND THEN CLEANED UP

update archi - questions

dl wsdl before or after node disco ?? BOTH cached

WebResourceDownloader (for pull) :

• see Resource Download Architecture
• TODO May be cached with timestamp (client-side, HTTP style)
• TODO with source probe id

OPT async Work tasks :

• impl alts : see https://github.com/easysoa/EasySOA/wiki/Long-running-tasks-&-workflows-in-nuxeo
• own itf, specific (updateWSDL(s)) or generic (executeTask()) ? remote, using WebEngine or Automation ??
• other tasks(-like) : ValidationTask...

TODO deployment :

• put WebResourceDownloader in HTTP Proxy ?
• put HTTP Proxy default instance in FStudio, rather than in standalone war ?
• TODO use alias conf'd in prop file to conf remote injection & addresses

TODO rest core outside nuxeo deps & pom

TODO LATER2 allow to create new auth domain in (registry and) WRDS by recording & registering (previous) login request(s) to allow replay

how is update triggered :

• most resources (records (samples...) by monit / driver, resource files by source disco) can't be updated by themselves => updating is a feature of the probe,
• each probe has its own update policy (auto schedule possibly according to registry conf, manual operation...)
• BUT NOT USING AUTOMATION (or other Nuxeo generic way) else no specific knowledge of update, OR automation then notify that it's been udpated by the probe (using metadata OR OPT WebEngine or automation also, possibly by throwing custom event triggering update, which is done using OPT Work)
• TODO wsdl auto update also provides endpoint availability ?!! though disco metas are on the doc, while state meta are in SQLDirectory
• TODO update endpoints, but not services because they are user-defined expectations => different update policies for both

TODO how to make it all scalable ?? ex. :

• rather pull (than push) or else SQL Directory (instead of doc metas) NOT FOR SOURCE DISCO
• async Work
• CEP/esper & disco aggregation...

web disco - wsdl archi alts

WSDL web resource download by (web) disco - alt 1 :

• NO TODO LATER rather only for scheduled update of service/endpoint interface or even web resource (on service/impl/endpoint, Phase, Component, query...)
• WsdlServiceDefinitionUpdate(r/)Task (extends abstract ServiceDefinitionUpdate(r/)Task) (OR more generic Resource(Download)Update(r/)Task)
• develop & configure it (see TODO wiki) so when run, it :
• (gets provided WSDL OR) downloads WSDL using url & timestamp through conf'd WebResourceDownloader,
• sets it on ex. iserv
• and triggers WsdlParsingListener OR BETTER by custom onServiceDefinitionDownloaded event (and only let iserv having "NuxeoUI" source probe meta trigger WsdlParsingListener through onDocumentUpdated) TODO AND POSSIBLY EVEN then matching (OR onServiceDefinitionParsed event)
• when doing this, IF NOT LONG decouple WSDL parsing from updating documentModel by coding Interface/WsdlService.getInterface/WsdlInfo, else LATER
• (web & monit) disco :
• called after SOA node discovery with DownloadedResourceInfo i.e. url, timestamp & source probe (instance id) of WSDL, as well as just registered ex. iserv
• by, either remotely probe (node proxy & HTTPProxy) using (web or monit) disco's WSDL's DownloadedResourceInfo
• TODO OR locally in DiscoveryService using those stored on ex. iserv, explicitly OR BETTER onNodeDiscovered event
• NB. who does / controls dl of wsdl : node (/ HTTPProxy) or nuxeo ?? => IN ANY WAY THE DBB PROBE PART at least by a "source probe" context param. & who does / controls parsing of wsdl : TODO idem ???
• TODO TODO therefore LATER refactor web disco-specific parts AND / OR WSDL-specific parts (WSDL parsing...) in its own nuxeo plugin(s)
• LATER scheduled update : uses url & timestamp stored on ex. iserv
• NB. source disco : would require Nuxeo to have a source manager (external but packaged with it, or embedded) providing source files locally, doesn't work with dev local / uncommitted work

WSDL web resource download by (web) disco - alt 2 (push) :

• probe sends disco'd SoaNodeInformations, LATER with probe metas
• probe sends resource file i.e. WSDL (dl'd again or cached from WebResourceDownload service), using 0,5 Automation OR 1,5 custom WebEngine BETTER FOR CLIENTS THAT ARE PURE JAXRS REST EX. source, monit BUT engine specific unless OPT 0,5 provide client custom multipart handling http://jsfprimefacesblog.blogspot.fr/2011/07/rest-fileupload-filedownload-example.html
• OPT and notifies of it (in same or separate call), by : 0,5 "(interface)contentChanged=true" meta in another disco OR 1 custom WebEngine (interface)EventRestService.notify((interface)contentChanged) OR 2 custom Automation
• listener or OPT 3 async Work extracts / parses / copies meta & matches.
• NB. if async ParsingWork (or DlWork), harder for probe to get a possible match ex. for displaying it (requires similar generated callback)
• LATER scheduled update : 2 async DlWork + notifies

alt 3 (pull beyond WRDS) :

• probe exposes the resource at a (0,5 permanent if WebResourceDownloadService OR) temporary URL (1 node, 2 FraSCAti or Jersey) ; using ex. random (TODO ??) temporary tickets (i.e. same as "by hand" async listener ; in memory table mapping to web or filesystem URLs, timeoutable), for which source probe id is useful
• 0,5 probe provides this URL within disco info
• OR 0,5 probe sends this non-local URL to the Registry, 1 which exposes sync DownloadRestService(Impl).doDownload(...) OR 3 async (Download)RestTaskService(Impl).schedule(Download)Task(DownloadedResourceInfo : url, downloadUrl, TODO docId or soaId ??, OPT timestamp, source probe (instance id) for routing) in REST which schedules a (Resource)DownloadTask, both using WebResourceDownloadService
• which downloads, sets and OPT 0,5 notifies WSDL for OPT 3 async Work parsing
• NB. pbs : firewall etc. when distant probes (=> WebResourceDownloadService is an answer to that for web resource files), harder for probe to get a possible match ex. for displaying it (requires similar generated callback)

web disco - wsdl archi comparison :

• wsdl web (n) : push 0,5-1,5 : na0,5 + (OPT 0,5 or r1 or a2) or nr1,5 ; pull 0,6-3 : d0,1 or jn1,5, 0,5 (OR + 1 or w3)
• wsdl monit (f, (h+json & a)) : push 0,5-2,5 : a0,5 + (OPT 0,5 or fr1 or a2) or fr1,5 + 1 ; pull 0,6-4 : d0,1 or jf2,5, 0,5 (OR + 1 or w3)
• wsdl etc. source (j, ((h,a))) : push 1-2 : a1 + (OPT 0,5 or jr1 or a2,5) or jr1,5 + 0,5 ; pull 2,5-3,5 : jj2, 0,5 (OR + 1 or w3)
• wsdl auto update (j) : pull 0,6-1,6 : d0,1, 0,5 (OR + 1 or w3)
• exchange sample :

cost for users to code a client in their own probe :

• cas automation (i.e. using our simple dedicated library) : push : 1
• cas REST (i.e. having to implement the remoting part of the probe, or reusing our j or f) : push : 2 ; pull d : d1 + 0,5 ; pull td : d1 + 4

dependencies of probes now :

• web : node
• monit : h+json & a (api-remote), FraSCAti
• source : jersey, TODO remove deps to nuxeo rest server & classification...
• NB. no more wsdl dl in web disco 2 because was dl'd from server
• NB. automation is built on Apache HTTP Client Component

disco beyond web wsdl

WSDL web resource download by monit disco :

• on server-side, same as web disco (with the more important case of on demand probes)
• use HTTP Proxy conf "projectId" as Phase id, possibly rename it to "subprojectId" or "phaseId"
• inject RegistryApi in FraSCAti, like in integration-base test
• update to new model :
• Service : put callcount (& participants ?) out OPT to state notif, LATER update these features
• remove others (Api & Appli => conf'd LATER Component, abstract Node, unused WSDLService), use RegistryApi instead
• remove now useless NuxeoRegistrationService and its org.easysoa.proxy.core.api.nuxeo.registration (which should have been migrated to easysoa-api-registry-remote but was not) (TODO OR put it in its own package or even module)

resource files by source disco :

• a resource file (ex. wsdl, service/impl/endpoint conf / policy) is sent by (source) disco probe, TODO through Disco, WebEngine, Automation ??
• it is created in the Registry as its own type or descendant type of a generic ResourceFileDocument, and its meta auto-extracted / parsed by its similarly typed listener, TODO on custom event ??
• TODO how is it linked to the SOA model ex. service/impl/endpoint ?? matching-like using extracted specific metas, else manually ? how pluggable ?
• update is done by probe, according to its configuration : ex. scheduled, on event (typically a Jenkins build) ; manual update must be done through the probe (ex. explicitly ask Jenkins to rebuild)
• TODO LATER refactor interfaces of service & endpoint to be in their own resource files as well, and auto-copy/sync their metas to service & endpoint

records (samples...) by monit / driver :

• an exchange record sample is a resource as well, sent by the monit / driver
• TODO who manages its (archive) file(s) ? the same who parses it / provides its meta ?? i.e. driver instance ??? and if on demand ????
• manager impl alts :
• Nuxeo (as content file(s)) ; OR external but then which :
• dedicated (as now : RunManagerDriverImpl+HTTPProxy on top of the filesystem of where it is instanciated ; TODO may be saved / sync'd using git ; TODO filesystem on demand FStudio ; TODO id share FStudio apps using git(hub) as well),
• or built on top of a central instance of source management ex. CI's,
• or of Nuxeo's own instance => NO would be the same as embedded CI therefore only a matter of packaging TODO DO IT WITH IMPROVE
• NB. TODO same for wsdls, the registry's wsdl (and version) is the reference, and the source code's is only the source of it ????
• LATER other types of samples : auto-extracted from JUnit by source disco, service + operation + params playable calls, manual description / wiki only...
• => TODO Nuxeo meta schema

web disco - new wsdl archi

summary (from xls) - Resource Update framework & for web disco :

• Define resource-update-info schema (RDI)
• Define & conf (Sync/AsyncWork)ResourceUpdateService, (Wsdl)ResourceParsingService
• Code SyncResourceUpdateService according to RDI, put probe conf in ProbeConfUtil (LATER Probe(Conf)Service)
• Copy WsdlParsingListener to ResourceParsingListener to do the start of the job (& same event hookup conf)
• refactor the next part of WsdlParsingListener to WsdlResourceParsingService and call it in ResourceParsingListener
• make it all async : end of ResourceUpdateService triggers resourceUpdatedEvent

resource download info schema :

• "actual" url, valid from download probe (instance) context (which includes network, auth i.e. WRDS...). May also be a file URL.
• OPT downloadable url, where download probe provides the resource. Optional because may stay hidden in probe's ResourceUpdater. Null if it doesn't. Null until inited by probe, which may be ex. WSDLParsingListener using the (OPT probe's) configured WebResourceDownloadService.
• timestamp of retrieval. If differs, triggers parsing if no dedicated notif con'd for the probe. If null, inited by the registry. May also be used for caching.
• probe (type) (instance) id. ? allows routing (meaning host/ip must be available, directly from id or mapped in a routing conf ex. genapp)
• NB. this schema on service or endpoint means WSDL, else (and LATER for service & endpoint) it depends on the document type : interface, policy, other conf...

probe conf :

• NB. for now hardcoded according to probe type (rdi.probe) in XXXResourceListener, when required refactored to ProbeService, LATER made configurable AND / OR even changeable dynamically (stored as docs & meta OR in its own "embedded ESB" (ex. FStudio genapp) db ?! OR both ?)
• isResourceProbeEventCustom(probe, event) (or getCustomResourceProbeEvent ?)
• isSelfUpdated (or get(Probe)RegistryResourceUpdater/get(Probe)WebResourceDownloadService() == null)
• get(Probe)RegistryResourceUpdater may
• return a Sync/SimpleUpdater or OPT an AsyncUpdater built on ResourceUpdaterWork
• use the WRDS provided by get(Probe)WebResourceDownloadService(), which may be the Registry's default one or a probe conf-specific one (typically, the WRDS hosted on the probe, whose ip/host is stored in its conf)

new WSDL disco & parsing archi :

• NB. what varies among Resources :
• how it is updated/downloaded : ResourceUpdatERService. According to probe conf, see above.
• parsed/extracted : ResourceParsingService. According to Resource precise content type, from its Document type but also iserv:interface(Definition)Type meta to support service/endpoint = Resource (LATER refactor to separate them)
• (what it extracts : ex. restinfo interface info may be extracted from both JAXRS & swagger interface definitions)
• what it triggers (ex. for Interfaces : MatchingService, to be refactored out of MatchingListener)
• InformationService & Endpoint changes (or on dedicated refreshWebResources ??? which & where ?) trigger (Wsdl/Interface)ResourceListener LATER extends ResourceListener, which :
• exits if not isResourceProbeEvent() i.e. if document change but probe conf (for now hardcoded, LATER conf) says it triggers using dedicated event
• exits if not isNewResourceRetrieval() i.e. if timestamp is non null & doesn't differ, exit
• get probe conf.download : if url not null update WSDL using (OPT probe's, by suppl param) WebResourceUpdatERService, OPT save if probe conf says it provides itself the WSDL then simply find it on doc
• WebResourceUpdatERService : provided with RDI as params, download and set WSDL OPT using AsyncWebResourceUpdatERService/Work triggering onWebResourceFileUpdated, init timestamp if non null OPT using AsyncWorkWRDS
• then OR on lower priority event OR OPT async on onWebResourceUpdated event, parse WSDL using WsdlParsingService, OPT triggering onWebResourceUpdated
• then OR on lower priority event OR OPT async on onWebResourceUpdated event, do matching

new WSDL monit disco :

• send also rdi:url to WSDL in monit disco, which triggers ResourceListener download, parsing (OPT & matching)
• Use global WRDS if any needed (made available in Fstudio, or else in ES Proxy if packaged in it)

new WSDL source disco archi :

• RDI info is also sent in disco (triggers ResourceListener but won't do anything because no URL)
• WSDL is sent using automation OR OPT custom WebEngine
• which triggers WSDL listener which does (OR triggers) parsing (OPT & matching)

new WSDL auto update archi :

• WebResourceUpdateJob (?) triggers refreshWebResources event with params (Phase(s) OR OPT (param'd) query) TODO see study
• configured to be scheduled