Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(xmlupload): refactor resource creation #917

Merged
merged 76 commits into from May 8, 2024
Merged
Show file tree
Hide file tree
Changes from 59 commits
Commits
Show all changes
76 commits
Select commit Hold shift + click to select a range
f7288e9
first step
jnussbaum Apr 22, 2024
3198944
typo
jnussbaum Apr 22, 2024
2fef922
factor out _upload_one_resource (primitive approach)
jnussbaum Apr 22, 2024
03902a5
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum Apr 23, 2024
8cb2912
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum Apr 24, 2024
b42fdca
add test file
jnussbaum Apr 24, 2024
d9b745b
create UploadState further above in the call stack
jnussbaum Apr 24, 2024
9ea7f46
beautify
jnussbaum Apr 24, 2024
fb48492
edit
jnussbaum Apr 24, 2024
08a0f2d
fix
jnussbaum Apr 24, 2024
a71dba5
UploadState contains IriResolver instead of dict
jnussbaum Apr 24, 2024
0886017
simplify
jnussbaum Apr 24, 2024
b422903
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum Apr 24, 2024
a2dac8a
revert unnecessary changes in test_upload_stash_with_mock
jnussbaum Apr 24, 2024
2b5ef27
simplify diff
jnussbaum Apr 24, 2024
2bb9ae3
one single try-except
jnussbaum Apr 24, 2024
d6fe625
fix docstring
jnussbaum Apr 24, 2024
d4b4f9b
first test (not working yet)
jnussbaum Apr 24, 2024
f945307
simplify code
jnussbaum Apr 25, 2024
6652720
fix first test
jnussbaum Apr 25, 2024
1934777
add another test
jnussbaum Apr 25, 2024
55cbf7e
refactor
jnussbaum Apr 25, 2024
290e8ad
test from 1 level higher
jnussbaum Apr 25, 2024
52e8292
refactor
jnussbaum Apr 25, 2024
c04f671
reformat
jnussbaum Apr 25, 2024
d4db53a
continue
jnussbaum Apr 25, 2024
85d5ea6
fix 3rd test
jnussbaum Apr 25, 2024
a268f09
4th test
jnussbaum Apr 25, 2024
bd19b06
add another test
jnussbaum Apr 25, 2024
557794e
new file + rename
jnussbaum Apr 25, 2024
09caba7
add unit tests for idempotency
jnussbaum Apr 25, 2024
df6e5d0
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum Apr 25, 2024
7143bd6
interrupt_after: still failing
jnussbaum Apr 25, 2024
350fe8f
fix attempt
jnussbaum Apr 25, 2024
8f8aec0
continue
jnussbaum Apr 26, 2024
464a08f
add warning that tidy up is in progress
jnussbaum Apr 26, 2024
10d1188
edit
jnussbaum Apr 26, 2024
f5ede09
fix?
jnussbaum Apr 26, 2024
bc7feec
fix + rename
jnussbaum Apr 29, 2024
8ebc154
fix
jnussbaum Apr 29, 2024
bcaf655
fix linter
jnussbaum Apr 29, 2024
2a1a422
factor out reusable part
jnussbaum Apr 29, 2024
03b0ad3
use spec_set instead of spec
jnussbaum Apr 29, 2024
c3ac5d9
refactor tests
jnussbaum Apr 29, 2024
88bcf15
use numbers in function names
jnussbaum Apr 29, 2024
68ad343
add test_5_resources_with_stash_and_interrupt_after_2
jnussbaum Apr 29, 2024
7e2e219
use deepcopy
jnussbaum Apr 29, 2024
48bc813
check num of calls
jnussbaum Apr 29, 2024
c414d00
fix last test
jnussbaum Apr 29, 2024
47efc0f
reraise keyboardinterrupt after tidy up
jnussbaum Apr 29, 2024
3e5945a
test post calls separately
jnussbaum Apr 29, 2024
e17666a
shorter variable names
jnussbaum Apr 29, 2024
c1295d2
add a test for logging
jnussbaum Apr 29, 2024
8fb7e45
don't return label
jnussbaum Apr 30, 2024
b8ca72c
compute numbers at lowest level
jnussbaum Apr 30, 2024
f91122e
one single conftest.py
jnussbaum Apr 30, 2024
1931823
fix logging test
jnussbaum Apr 30, 2024
9ebf19c
test only INFO level
jnussbaum Apr 30, 2024
7f86b09
reset pre-commit config
jnussbaum Apr 30, 2024
f3ae4fa
docstring
jnussbaum Apr 30, 2024
ca77bfa
align warning class with Nora's feature branch
jnussbaum Apr 30, 2024
d6da780
fix import
jnussbaum Apr 30, 2024
8d827be
add docstring to conftest.py
jnussbaum Apr 30, 2024
51edfd6
ignore conftest.py file in distribution tests
jnussbaum Apr 30, 2024
5116e0a
make darglint happy
jnussbaum Apr 30, 2024
68fff6e
rename
jnussbaum Apr 30, 2024
1af1d49
make sure that only XmlUploadInterruptedError is raised
jnussbaum Apr 30, 2024
8689d34
cosmetics
jnussbaum Apr 30, 2024
c970197
rearrange test functions
jnussbaum Apr 30, 2024
b3e01ce
upload_resources(): don't return the nonapplied stash
jnussbaum Apr 30, 2024
a6af7a1
refactor + fix cleanup_upload()
jnussbaum Apr 30, 2024
9b5addb
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum Apr 30, 2024
ce78f61
refactor resume_xmlupload()
jnussbaum Apr 30, 2024
15e55e8
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum May 2, 2024
f2b66d2
Merge branch 'main' into wip/DEV-3522-refactor-resource-creation
jnussbaum May 8, 2024
dfcbdea
add inline comment
jnussbaum May 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
15 changes: 4 additions & 11 deletions src/dsp_tools/commands/resume_xmlupload/resume_xmlupload.py
Expand Up @@ -6,7 +6,6 @@
from loguru import logger
from termcolor import colored

from dsp_tools.commands.xmlupload.iri_resolver import IriResolver
from dsp_tools.commands.xmlupload.list_client import ListClient
from dsp_tools.commands.xmlupload.list_client import ListClientLive
from dsp_tools.commands.xmlupload.models.sipi import Sipi
Expand Down Expand Up @@ -52,7 +51,7 @@ def resume_xmlupload(server: str, user: str, password: str, sipi: str, skip_firs
if resp == "n":
sys.exit(1)

previous_successful = len(upload_state.iri_resolver_lookup)
previous_successful = len(upload_state.iri_resolver.lookup)
previous_failed = len(upload_state.failed_uploads)
previous_total = previous_successful + previous_failed
msg = (
Expand All @@ -72,21 +71,15 @@ def resume_xmlupload(server: str, user: str, password: str, sipi: str, skip_firs
project_client: ProjectClient = ProjectClientLive(con, upload_state.config.shortcode)
list_client: ListClient = ListClientLive(con, project_client.get_project_iri())

iri_resolver, failed_uploads, nonapplied_stash = upload_resources(
resources=upload_state.pending_resources,
failed_uploads=upload_state.failed_uploads,
nonapplied_stash = upload_resources(
upload_state=upload_state,
imgdir=".",
sipi_server=sipi_server,
permissions_lookup=upload_state.permissions_lookup,
con=con,
stash=upload_state.pending_stash,
config=upload_state.config,
project_client=project_client,
list_client=list_client,
iri_resolver=IriResolver(upload_state.iri_resolver_lookup),
)

return cleanup_upload(iri_resolver, upload_state.config, failed_uploads, nonapplied_stash)
return cleanup_upload(upload_state.iri_resolver, upload_state.config, upload_state.failed_uploads, nonapplied_stash)


def _read_upload_state_from_disk(server: str) -> UploadState:
Expand Down
5 changes: 3 additions & 2 deletions src/dsp_tools/commands/xmlupload/models/upload_state.py
@@ -1,20 +1,21 @@
from dataclasses import dataclass

from dsp_tools.commands.xmlupload.iri_resolver import IriResolver
from dsp_tools.commands.xmlupload.models.permission import Permissions
from dsp_tools.commands.xmlupload.models.xmlresource import XMLResource
from dsp_tools.commands.xmlupload.stash.stash_models import Stash
from dsp_tools.commands.xmlupload.upload_config import UploadConfig


@dataclass(frozen=True)
@dataclass
class UploadState:
"""
Save the state of an xmlupload, so that after an interruption, it can be resumed.
"""

pending_resources: list[XMLResource]
failed_uploads: list[str]
iri_resolver_lookup: dict[str, str]
iri_resolver: IriResolver
pending_stash: Stash | None
config: UploadConfig
permissions_lookup: dict[str, Permissions]
4 changes: 4 additions & 0 deletions src/dsp_tools/commands/xmlupload/project_client.py
Expand Up @@ -19,6 +19,10 @@ class ProjectInfo:
class ProjectClient(Protocol):
"""Interface (protocol) for project-related requests to the DSP-API."""

con: Connection
shortcode: str
project_info: ProjectInfo | None

jnussbaum marked this conversation as resolved.
Show resolved Hide resolved
def get_project_iri(self) -> str:
"""Get the IRI of the project to which the data is being uploaded."""

Expand Down
7 changes: 3 additions & 4 deletions src/dsp_tools/commands/xmlupload/resource_create_client.py
Expand Up @@ -3,6 +3,7 @@
from pathlib import Path
from typing import Any
from typing import assert_never
from typing import cast

from loguru import logger

Expand Down Expand Up @@ -37,17 +38,15 @@ def create_resource(
self,
resource: XMLResource,
bitstream_information: BitstreamInfo | None,
) -> tuple[str, str]:
) -> str:
"""Creates a resource on the DSP server."""
logger.info(
f"Attempting to create resource {resource.res_id} (label: {resource.label}, iri: {resource.iri})..."
)
resource_dict = self._make_resource_with_values(resource, bitstream_information)
headers = {"X-Asset-Ingested": "true"} if self.media_previously_ingested else None
res = self.con.post(route="/v2/resources", data=resource_dict, headers=headers)
iri = res["@id"]
label = res["rdfs:label"]
return iri, label
return cast(str, res["@id"])

def _make_resource_with_values(
self,
Expand Down
6 changes: 6 additions & 0 deletions src/dsp_tools/commands/xmlupload/stash/stash_models.py
Expand Up @@ -106,3 +106,9 @@ def make(standoff_stash: StandoffStash | None, link_value_stash: LinkValueStash
if standoff_stash or link_value_stash:
return Stash(standoff_stash, link_value_stash)
return None

def is_empty(self) -> bool:
"""Check if there are any stashed items in this stash"""
standoff = not self.standoff_stash or not self.standoff_stash.res_2_stash_items
link = not self.link_value_stash or not self.link_value_stash.res_2_stash_items
return standoff and link