v0.1 #1

expede · 2022-12-01T06:42:25Z

expede · 2022-12-01T07:05:57Z

README.md

+{
+  "with": "ipfs://bafkreidvq3uqoxcxr44q5qhgdk5zk6jvziipyxguirqa6tkh5z5wtpesva",
+  "can": "docker/run",
+  "inputs": {
+    "entrypoint": ["/", "home"],
+    "mounts": {"mnt": "/"},
+    "outputs": {"tmp": "/tmp"},
+    "env":  {"$HELLO": "world"},
+    "workdir":  "/work"
+  },
+  "meta": {
+    "ipvm/config": {
+      "secret": false,
+      "check": {
+        "optimistic": 2, 
+        "referee": "did:key:zStEZpzSMtTt9k2vszgvCwF4fLQQSyA15W5AQ4z3AR6Bx4eFJ5crJFbuGxKmbma4"
+      }
+    }
+  }
+}
+```
+
+``` json
+{
+  "with": {"ucan/promise": ["/", "previous_action"]},
+  "can": "docker/run",
+  "inputs": {
+    "entrypoint": ["/", "home"],
+    "mounts": {"mnt": "/"},
+    "outputs": {"tmp": "/tmp"},
+    "env":  {"$HELLO": "world"},
+    "workdir":  "/work"
+  },
+  "meta": {
+    "ipvm/config": {
+      "disk": "200GB"
+    }
+  }
+}
+```


From comments on the old repo, I've included examples of how to both pull a Docker container off the network & run it, as well as how to do this with a pipeline.

(fixed s/can/do/ 😅 )

zeeshanlakhani

First run on comments. Will have more after our discussion(s).

README.md

zeeshanlakhani · 2022-12-01T15:14:12Z

README.md

+
+``` ipldsch
+type SystemConfig struct {
+  secret Boolean      (implicit False)


instead of implicit should these say defaults or defaults to as more typical?

It's an option in IPLD Schema https://ipld.io/docs/schemas/features/typekinds/#value-type-modifiers

The other note on IPLD Schema is that it's an atypical system that mixes types and representation pretty liberally. I'm still learning the ins-and-outs myself.

zeeshanlakhani · 2022-12-01T15:16:38Z

README.md

+| `inputs` | `Any`            |                                                | Yes      |         |
+| `meta`   | `{String : Any}` | Fields that will be ignored during memoization | No       | `{}`    |
+
+An OPTIONAL IPVM `Config` MAY be included at the `meta['ipvm/config']` path. The `meta` field SHOULD not captured as part of task memoization, so this information will be omitted from the distributed invocation table. If included, the `Config` MUST set the IPVM configuration for this Task, overwriting any of the fields on the envelope's top-level `defaults` field, or system-wide defaults.


meta feels much more optional than config. Shouldn't config and/or something like turning on/off certain features be its own field, as it seems less "throw-away" than meta?

zeeshanlakhani · 2022-12-01T17:04:55Z

README.md

+| `secret` | `Boolean or null` | Whether the output is unsafe to publish | No       | `null`                   |
+| `check`  | `Verification`    | Verification strategy                   | No       | `"attestation"`          |
+| `time`   | `TimeInterval`    | Timeout                                 | No       | `[5, "minutes"]`         |
+| `memory` | `InfoSize`        | Memory limit                            | No       | `[100, "kilo", "bytes"]` |


Q: on infosizes, would it be better to specify units like kb, ms, as that can be enumerated upon?

Haha everyone hates it when I do this, but I keep trying in every project 😆

can be enumerated upon?

It is enumerated! Just composable, so we don't need kB, MB, TB, PB, ns, us, ms, etc. I guess it's not a huge deal either way. We could also set these fields to string.

If you still dislike it I can pull it out.

Yeah. no biggie. I've been used to systems like prom, etc, which have:

Units supported: B, KB, MB, GB, TB, PB, EB. Ex: "512MB". Based on powers-of-2, so 1KB is 1024B

So, are people more used to that, maybe?

You're right — these kinds of libraries are very common and pretty much what people expect. Ugh, stringly typed things that I have to parse a bunch of different ways (whitespace etc) is gross 😆

100% of the time that I've done this someone has mentioned it, so clearly it's against the principle of least surprise. Will revert!

I love it, please keep it! 😅

zeeshanlakhani · 2022-12-01T17:12:20Z

README.md

+| Field           | Type       | Description                                                             | Required |
+|-----------------|------------|-------------------------------------------------------------------------|----------|
+| `ipvm/workflow` | `Workflow` | IPVM Workflow                                                           | Yes      |
+| `signature`     | `VarSig`   | [VarSig](https://github.com/ChainAgnostic/varsig/) of serialized fields | Yes      |


Seems like the repo is still WIP for VarSig? Can we get a footnote about it?

We're trying to retcon varsigs a bit by adding more data to them. I hope to get a spec out in the next day or two.

zeeshanlakhani · 2022-12-01T17:14:50Z

README.md

+|------------|------------------------------|------------------------------------------------------------------------|----------|---------|
+| `v`        | `"0.1.0"`                    | IPVM workflow version                                                  | Yes      |         |
+| `meta`     | `{String : Any}`             | User-defined object (tags, comments, etc)                              | No       | `{}`    |
+| `parent`   | `[&Workflow, Label] or Null` | The workflow & task label that initiated the current workflow (if any) | No       | `Null`  |


Minus general provenance/lineage, @expede how do you see parent being used in an implementation of workflow(s)?

Entirely for provenance & tracing

README.md

zeeshanlakhani · 2022-12-01T17:34:25Z

README.md

+| `config`   | `Config`                     | Global configuration (e.g. timeout for the entire workflow)            | No       | `{}`    |
+| `defaults` | `Config`                     | Individual task config defaults                                        | No       | `{}`    |
+| `tasks`    | `UCAN.Invocation`            | UCAN Invocation                                                        | Yes      |         |
+| `catch`    | `&WasmTask`                  | Deterministic Wasm that fires on exceptions                            | No       | `{}`    |


so is wasmtask a ucan/invocation task, as it defaults to a {}? I ask because elsewhere in the doc, it looks like a CID directly. I wonder if catch points to a CID or wasm::CID and/or to a known top-level key/name of an invocation task that may already have been introduced previously?

So, here catch is wasm + determinism-specific, so it should not be allowed on the workflow top-level if an inner task is a docker one.

as it defaults to a {}

Oops! That should definitely default to Null. Will fix!

a known top-level key/name of an invocation task

I want to keep it in a separate block for a couple reasons:

Pure Wasm only

Set really tight, enforced resource restrictions (it should terminate quickly)

If it's a named task, it will get scheduled with the rest of the workflow

I need to tighten up some of these, too. That really should be &Wasm.

zeeshanlakhani · 2022-12-01T17:46:58Z

README.md

+
+Note that while IPVM MUST treat the pure tasks together as transactional. It is not possible to roll back any destructive effects that have already been run. As such, it is RECOMMENDED to have few (if any) tasks depend on the output of a destructive effect, so they can be scheduled at the end of the workflow.
+
+# 6 Receipt Output


Personally, I'd love to see an example of the receipt output in json and maybe linked through with a workflow example. It'd also be great to link the result CID w/ label to another, new workflow?

Great idea! Per our conversation after you wrote this comment, we'll put together an examples directory 👍

zeeshanlakhani · 2022-12-01T17:49:03Z

README.md

+
+IPVM workflows are built on top of cryptographic capabilities, providing a strong basis for distributed computation in trustless networks. This even provides a clear basis for crossing Web 2.0 and Web3 systems, other computation networks, local operation (in full or part).
+
+The IPVM Workflow spec extends on several other specs that have been developed to provide this basis:


I know the frontmatter still uses job (i.e. https://github.com/ipvm-wg/spec), but it may be good to highlight why we're using workflow > job, especially when other systems like Bacalhau use job as well.

I know the frontmatter still uses job (i.e. https://github.com/ipvm-wg/spec)

Great point! It will not use the term "job" in the long-term. Consider that old text, which will get updated independently.

zeeshanlakhani · 2022-12-01T19:00:50Z

README.md

+
+If the `msg` branch is returned, the invocation MUST immediately rethrow with the update message.
+
+Note that while IPVM MUST treat the pure tasks together as transactional. It is not possible to roll back any destructive effects that have already been run. As such, it is RECOMMENDED to have few (if any) tasks depend on the output of a destructive effect, so they can be scheduled at the end of the workflow.


any thoughts on how to enforce this in any way?

I've been thinking about destructive actions almost like a cut in Prolog. You lose transactional guarantees as soon as you use them. We may want to include a flag or something to set your expected properties like transactional, but that's also probably a big ask of most devs. Handling this in tooling with e.g. a warning like "hey, just so you know, we won't be able to roll back prior to this email if something goes wrong"

zeeshanlakhani · 2022-12-01T19:02:42Z

README.md

+| `out`  | `{String: Any}` | The results of each call, the task's label. MAY contain sub-receipts. | Yes      |         |
+| `meta` | `Any`           | Non-normative extended fields                                         | No       | `null`  |
+
+If the `catch` field is set on the outer `Workflow`, The `out` field MAY include the output under the `ipvm/catch` key


would we have a need to actively want to throw an exception based on the execution of a task?

We also covered this on the call after this was posted. For posterity:

I've been thinking about this as akin to stdout and stderr. If you run an HTTP PUT, and it fails, the runtime needs to return that as a Failure. If a Docker container exits with a nonzero exit code, we'll capture its stderr and put that in the Failure struct.

Good point: this needs to be in the text 👍

zeeshanlakhani · 2022-12-01T20:33:03Z

README.md

+>
+> J. Paul Morrison, [Flow-Based Programming](https://jpaulm.github.io/fbp/book.html)
+
+The potential complexity of a fully distributed execution by untrusted peers is very high. IPVM Workflows reduce the number of possible states by forcing explicit handling of any dangerous effects. The IPVM Workflow spec is a declarative document that MAY be inspected, transmitted, logged, and negotiated. Unlike s systems like WASI, there is a strict separation of effects from pure data, an emphasis on verifiability, and [promise pipelining](http://erights.org/elib/distrib/pipeline.html).


Q: are effects still getting their own write-up?

(For those following along at home: Zeeshan and I chatted about this live)

They are, absolutely! I'm trying to keep the various specs as separate as possible; my current hypothesis is that we're going to do the scheduler safety properties for URI/Ability pairs as part of service discovery. They'll need to register something like:

Service { uri_scheme = "https" , ability = "crud/read" , idempotent = False , destructive = False , max_verifiability = Attestation }

We know that everything is some kind of effect unless it's raw Wasm, so that makes it really easy for the runtime to detect. If you want to run e.g. a pure subset of Python, you can definitely get an interpreter written in Wasm to feed the source into. We could open up pure: Bool as well, but the tooling for Wasm is advancing so quickly that I currently don't believe that we'll need anything beyond "run the source inside Wasm"

Co-authored-by: Zeeshan Lakhani <zeeshan.lakhani@gmail.com>

simonwo · 2022-12-02T16:05:59Z

README.md

+
+``` js
+{
+  "some-wasm": {


Note that these are shown in {String:Task} form – would it be clearer to just show the Task, to avoid confusion that the string is part of that?

expede changed the title ~~v0.1 draft~~ v0.1 Dec 1, 2022

expede marked this pull request as draft December 1, 2022 06:45

expede mentioned this pull request Dec 1, 2022

Job Spec ipvm-wg/spec#8

Draft

expede commented Dec 1, 2022

View reviewed changes

expede added 9 commits November 30, 2022 23:10

v0.1 draft

6db1689

Add actions

789e3d5

Set up CLA bot & spellcheck config

a42bcc7

Fix much spelling

0c6ec44

Spelling

d17f842

Remove JSON comment

151ea7e

Update Docker exmaple

c5e7ea0

Add to dict

3d7ddfd

Consisstency in examples

67aefef

expede force-pushed the v0.1 branch from 6205cd7 to 67aefef Compare December 1, 2022 07:10

cla-bot bot added the cla-signed label Dec 1, 2022

expede added 2 commits November 30, 2022 23:12

JSON typo

5e56970

Better layout for a couple JSON examples

4a8a14c

expede marked this pull request as ready for review December 1, 2022 07:16

zeeshanlakhani reviewed Dec 1, 2022

View reviewed changes

expede and others added 2 commits December 2, 2022 00:45

Update README.md

6d56430

Co-authored-by: Zeeshan Lakhani <zeeshan.lakhani@gmail.com>

Fix numbering

7e8e0e0

simonwo reviewed Dec 2, 2022

View reviewed changes

Update varsig spelling

993447c

zeeshanlakhani mentioned this pull request Jun 21, 2023

Spec Integration: Global workflow and task settings ⚙️ ipvm-wg/homestar#145

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v0.1 #1

v0.1 #1

expede commented Dec 1, 2022 •

edited

expede Dec 1, 2022

expede Dec 1, 2022

zeeshanlakhani left a comment

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 2, 2022

zeeshanlakhani Dec 1, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022 •

edited

zeeshanlakhani Dec 2, 2022

expede Dec 2, 2022

simonwo Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022

zeeshanlakhani Dec 1, 2022

expede Dec 2, 2022 •

edited

simonwo Dec 2, 2022


		Note that while IPVM MUST treat the pure tasks together as transactional. It is not possible to roll back any destructive effects that have already been run. As such, it is RECOMMENDED to have few (if any) tasks depend on the output of a destructive effect, so they can be scheduled at the end of the workflow.

		# 6 Receipt Output


		IPVM workflows are built on top of cryptographic capabilities, providing a strong basis for distributed computation in trustless networks. This even provides a clear basis for crossing Web 2.0 and Web3 systems, other computation networks, local operation (in full or part).

		The IPVM Workflow spec extends on several other specs that have been developed to provide this basis:


		If the `msg` branch is returned, the invocation MUST immediately rethrow with the update message.

		Note that while IPVM MUST treat the pure tasks together as transactional. It is not possible to roll back any destructive effects that have already been run. As such, it is RECOMMENDED to have few (if any) tasks depend on the output of a destructive effect, so they can be scheduled at the end of the workflow.

v0.1 #1

Are you sure you want to change the base?

v0.1 #1

Conversation

expede commented Dec 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zeeshanlakhani left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

expede Dec 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

expede Dec 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

expede commented Dec 1, 2022 •

edited

expede Dec 2, 2022 •

edited

expede Dec 2, 2022 •

edited