Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

上传过大（500MB+）的测试样例 Upload failed && 题目文件导入失败 #455

Open
Estrella-Explore opened this issue Mar 9, 2024 · 3 comments
Open

上传过大（500MB+）的测试样例 Upload failed && 题目文件导入失败 #455

Estrella-Explore opened this issue Mar 9, 2024 · 3 comments

Comments

@Estrella-Explore
Copy link

在提交issue之前请

  • 认真阅读文档 http://docs.onlinejudge.me/#/
  • 搜索和查看历史issues
  • 安全类问题请不要在 GitHub 上公布，请发送邮件到 admin@qduoj.com，根据漏洞危害程度发送红包感谢。

然后提交issue请写清楚下列事项

 - 进行什么操作的时候遇到了什么问题，最好能有复现步骤
 - 错误提示是什么，如果看不到错误提示，请去data文件夹查看相应log文件。大段的错误提示请包在代码块标记里面。

  • 你尝试修复问题的操作
  • 页面问题请写清浏览器版本，尽量有截图

复现操作：上传一个较大的数据点（测试样例文件）（约 500MB+）：

Upload failed.

截图

./data/backend/log/nginx.log：

2024/03/09 05:23:04 [warn] 106#106: *18746 a client request body is buffered to a temporary file /var/lib/nginx/tmp/client_body/0000000009, client: 61.136.xxx.xxx, server: _, request: "POST /api/admin/test_case HTTP/1.1", host: "47.115.xxx.xxx", referrer: "http://47.115.xxx.xxx/admin/problem/create"
2024/03/09 05:24:07 [error] 106#106: *18759 client intended to send too large body: 401714833 bytes, client: 61.136.xxx.xxx, server: _, request: "POST /api/admin/test_case HTTP/1.1", host: "47.115.xxx.xxx", referrer: "http://47.115.xxx.xxx/admin/problem/create"
2024/03/09 05:24:56 [error] 105#105: *18775 client intended to send too large body: 401714830 bytes, client: 61.136.xxx.xxx, server: _, request: "POST /api/admin/test_case HTTP/1.1", host: "47.115.xxx.xxx", referrer: "http://47.115.xxx.xxx/admin/problem/create"

另一个问题：
导入题目失败：

复现操作：导出题目后题目文件不加改动重新导入，有如下报错：

Invalid problem format, error is {'template': {'C++': {'prepend': [ErrorDetail(string='This field may not be blank.', code='blank')], 'template': [ErrorDetail(string='This field may not be blank.', code='blank')], 'append': [ErrorDetail(string='This field may not be blank.', code='blank')]}}}

./data/backend/log/gunicron.log：（log 提示没有 1/problem.json，但此压缩包中确实存在）

[2024-03-07 09:19:09 +0000] [120] [INFO] Starting gunicorn 20.1.0
[2024-03-07 09:19:09 +0000] [120] [INFO] Listening at: http://127.0.0.1:8080 (120)
[2024-03-07 09:19:09 +0000] [120] [INFO] Using worker: gthread
[2024-03-07 09:19:09 +0000] [124] [INFO] Booting worker with pid: 124
[2024-03-07 09:19:09 +0000] [125] [INFO] Booting worker with pid: 125
[2024-03-08 05:28:23 +0000] [120] [INFO] Handling signal: term
[2024-03-08 05:28:23 +0000] [124] [INFO] Worker exiting (pid: 124)
[2024-03-08 05:28:24 +0000] [125] [INFO] Worker exiting (pid: 125)
[2024-03-08 05:28:24 +0000] [120] [INFO] Shutting down: Master
[2024-03-08 05:29:03 +0000] [103] [INFO] Starting gunicorn 20.1.0
[2024-03-08 05:29:03 +0000] [103] [INFO] Listening at: http://127.0.0.1:8080 (103)
[2024-03-08 05:29:03 +0000] [103] [INFO] Using worker: gthread
[2024-03-08 05:29:03 +0000] [107] [INFO] Booting worker with pid: 107
[2024-03-08 05:29:03 +0000] [108] [INFO] Booting worker with pid: 108
[2024-03-08 05:56:56] - [ERROR] - [root:156]  - "There is no item named '1/problem.json' in the archive"
Traceback (most recent call last):
  File "/app/utils/api/api.py", line 149, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/app/problem/views/admin.py", line 571, in post    with zip_file.open(f"{i}/problem.json") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/usr/local/lib/python3.12/zipfile/__init__.py", line 1595, in open     zinfo = self.getinfo(name) 
  File "/usr/local/lib/python3.12/zipfile/__init__.py", line 1524, in getinfo
    raise KeyError(
KeyError: "There is no item named '1/problem.json' in the archive"
[2024-03-08 05:56:57] - [ERROR] - [sentry.errors:686]  - Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.12/site-packages/raven/transport/http.py", line 38, in send
    response = urlopen(
               ^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 37, in connect
    self.sock = ssl.wrap_socket(
                ^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'wrap_socket'
b"Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)"
[2024-03-08 05:56:57] - [ERROR] - [sentry.errors.uncaught:712]  - ['"There is no item named \'1/problem.json\' in the archive"', '  File "utils/api/api.py", line 149, in dispatch', '  File "django/views/generic/base.py", line 98, in dispatch', '  File "problem/views/admin.py", line 571, in post', '  File "zipfile/__init__.py", line 1595, in open', '  File "zipfile/__init__.py", line 1524, in getinfo']
b'[\'"There is no item named \\\'1/problem.json\\\' in the archive"\', \'  File "utils/api/api.py", line 149, in dispatch\', \'  File "django/views/generic/base.py", line 98, in dispatch\', \'  File "problem/views/admin.py", line 571, in post\', \'  File "zipfile/__init__.py", line 1595, in open\', \'  File "zipfile/__init__.py", line 1524, in getinfo\']'
[2024-03-08 05:57:02] - [ERROR] - [root:156]  - not well-formed (invalid token): line 1, column 2
Traceback (most recent call last):
  File "/app/utils/api/api.py", line 149, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/problem/views/admin.py", line 683, in post
    problems = FPSParser(tf.name).parse()
               ^^^^^^^^^^^^^^^^^^
  File "/app/fps/parser.py", line 15, in __init__
    self._etree = ET.parse(fps_path).getroot()
                  ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1204, in parse
    tree.parse(source, parser)
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 569, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: not well-formed (invalid token): line 1, column 2
[2024-03-08 05:57:02] - [ERROR] - [sentry.errors:686]  - Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.12/site-packages/raven/transport/http.py", line 38, in send
    response = urlopen(
               ^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 515, in open
    response = self._open(req, data)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 532, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 492, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/urllib/request.py", line 1344, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/local/lib/python3.12/http/client.py", line 1331, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1377, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1326, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.12/http/client.py", line 1085, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.12/http/client.py", line 1029, in send
    self.connect()
  File "/usr/local/lib/python3.12/site-packages/raven/utils/http.py", line 37, in connect
    self.sock = ssl.wrap_socket(
                ^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'wrap_socket'
b"Sentry responded with an error: module 'ssl' has no attribute 'wrap_socket' (url: https://sentry.io/api/263057/store/)"
[2024-03-08 05:57:02] - [ERROR] - [sentry.errors.uncaught:712]  - ['not well-formed (invalid token): line 1, column 2', '  File "utils/api/api.py", line 149, in dispatch', '  File "django/views/generic/base.py", line 98, in dispatch', '  File "problem/views/admin.py", line 683, in post', '  File "fps/parser.py", line 15, in __init__', '  File "xml/etree/ElementTree.py", line 1204, in parse', '  File "xml/etree/ElementTree.py", line 569, in parse']
b'[\'not well-formed (invalid token): line 1, column 2\', \'  File "utils/api/api.py", line 149, in dispatch\', \'  File "django/views/generic/base.py", line 98, in dispatch\', \'  File "problem/views/admin.py", line 683, in post\', \'  File "fps/parser.py", line 15, in __init__\', \'  File "xml/etree/ElementTree.py", line 1204, in parse\', \'  File "xml/etree/ElementTree.py", line 569, in parse\']'
[2024-03-08 05:57:07] - [ERROR] - [root:156]  - not well-formed (invalid token): line 1, column 2
Traceback (most recent call last):
  File "/app/utils/api/api.py", line 149, in dispatch
    return super(APIView, self).dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/problem/views/admin.py", line 683, in post
    problems = FPSParser(tf.name).parse()
               ^^^^^^^^^^^^^^^^^^
  File "/app/fps/parser.py", line 15, in __init__
    self._etree = ET.parse(fps_path).getroot()
                  ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 1204, in parse
    tree.parse(source, parser)
  File "/usr/local/lib/python3.12/xml/etree/ElementTree.py", line 569, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
@SongJunxi2000
Copy link

您好，请问上传较大数据点会失败后续有解决掉吗？我们目前遇到的问题是20MB的数据点都无法上传。谢谢

@Estrella-Explore
Copy link
Author

您好，请问上传较大数据点会失败后续有解决掉吗？我们目前遇到的问题是20MB的数据点都无法上传。谢谢
@SongJunxi2000

Nope.

@zhblue
Copy link

zhblue commented May 22, 2024

如果测试数据是放在特定目录里的，可以考虑用sftp等方式上传。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zhblue @SongJunxi2000 @Estrella-Explore