python 如何将大数据流式传输到Google Cloud Storage?

egdjgwm8  于 2023-06-04  发布在  Python
关注(0)|答案(2)|浏览(165)

我正在开发一个系统,用于分析用户流到我的基于Google Cloud Storage的私有云的任何大小和格式的数据。你有什么想法,我如何让他们流大数据?目前我使用Django API,我是这样做的:

def upload_blob(source_file_name, destination_blob_name):
    blob = bucket.blob(destination_blob_name)
    blob.upload_from_filename(source_file_name)
    print('File {} uploaded to {}.'.format(
        source_file_name,
        destination_blob_name))

它可以正确地处理小文件,但是当我发送例如大电影时,我会得到下面显示的错误。我知道这不是最佳解决方案,但我不知道如何解决这个问题。正如你现在所注意到的,他们向我发送的请求是blob格式的,但是对于非常大的文件,它不起作用。你有什么想法,我如何解决我的问题,并发送任何大小的用户数据到谷歌云存储?

Internal Server Error: /cloud/
Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
  chunked=chunked,
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
  conn.request(method, url, **httplib_request_kw)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
  self._send_request(method, url, body, headers, encode_chunked)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1298, in _send_request
  self.endheaders(body, encode_chunked=encode_chunked)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1247, in endheaders
  self._send_output(message_body, encode_chunked=encode_chunked)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1065, in _send_output
  self.send(chunk)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 987, in send
  self.sock.sendall(data)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1034, in sendall
  v = self.send(byte_view[count:])
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1003, in send
  return self._sslobj.write(data)
socket.timeout: The write operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
  timeout=timeout
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
  method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
 File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
  raise six.reraise(type(error), error, _stacktrace)
 File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
  raise value.with_traceback(tb)
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
  chunked=chunked,
 File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
  conn.request(method, url, **httplib_request_kw)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1252, in request
  self._send_request(method, url, body, headers, encode_chunked)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1298, in _send_request
  self.endheaders(body, encode_chunked=encode_chunked)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1247, in endheaders
  self._send_output(message_body, encode_chunked=encode_chunked)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 1065, in _send_output
  self.send(chunk)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py", line 987, in send
  self.sock.sendall(data)
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1034, in sendall
  v = self.send(byte_view[count:])
 File "/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ssl.py", line 1003, in send
  return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/django/core/handlers/exception.py", line 34, in inner
  response = get_response(request)
 File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 115, in _get_response
  response = self.process_exception_by_middleware(e, request)
 File "/usr/local/lib/python3.7/site-packages/django/core/handlers/base.py", line 113, in _get_response
  response = wrapped_callback(request, *callback_args, **callback_kwargs)
 File "/usr/local/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
  return view_func(*args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/django/views/generic/base.py", line 71, in view
  return self.dispatch(request, *args, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 505, in dispatch
  response = self.handle_exception(exc)
 File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 465, in handle_exception
  self.raise_uncaught_exception(exc)
 File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 476, in raise_uncaught_exception
  raise exc
 File "/usr/local/lib/python3.7/site-packages/rest_framework/views.py", line 502, in dispatch
  response = handler(request, *args, **kwargs)
 File "/mypath/backend/views.py", line 635, in post
  'user/' + str(user_name) + '/' + str(file))
 File "/mypath/backend/views.py", line 214, in upload_blob
  blob.upload_from_filename(source_file_name)
 File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1318, in upload_from_filename
  predefined_acl=predefined_acl,
 File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1263, in upload_from_file
  client, file_obj, content_type, size, num_retries, predefined_acl
 File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1173, in _do_upload
  client, stream, content_type, size, num_retries, predefined_acl
 File "/usr/local/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1120, in _do_resumable_upload
  response = upload.transmit_next_chunk(transport)
 File "/usr/local/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 425, in transmit_next_chunk
  retry_strategy=self._retry_strategy,
 File "/usr/local/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136, in http_request
  return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
 File "/usr/local/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150, in wait_and_retry
  response = func()
 File "/usr/local/lib/python3.7/site-packages/google/auth/transport/requests.py", line 216, in request
  method, url, data=data, headers=request_headers, **kwargs
 File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
  resp = self.send(prep, **send_kwargs)
 File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
  r = adapter.send(request, **kwargs)
 File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
  raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))
[24/Mar/2020 19:17:26] "POST /cloud/ HTTP/1.1" 500 20879
am46iovg

am46iovg1#

看看Resumable uploads
此选项提供可恢复的数据传输功能,使您可以在通信故障中断数据流后恢复上载操作。
在传输大型文件时特别有用,因为网络中断或其他传输故障的可能性很高。在失败的情况下,使用此选项时,您不必从头开始重新启动大文件上载。

vaj7vani

vaj7vani2#

如果您仔细查看错误消息,您可以看到原因是超时。默认情况下,超时时间为60秒,但您可以指定更长的时间。blob.upload_from_filename(f'{filename}', timeout=300)等待300秒(或5分钟)。

相关问题