tensorflow AWS Sagemaker是否支持gRPC预测请求?

92dk7w1h  于 2022-11-16  发布在  其他
关注(0)|答案(1)|浏览(135)

我在本地模式下从估计器部署了Sagemaker的Tensorflow模型,当尝试使用gRPC调用Tensorflow Serving(TFS)预测端点时,我得到错误:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"

我正在执行gRPC请求,与此blog post中的完全相同:

import grpc from tensorflow.compat.v1 
import make_tensor_protofrom tensorflow_serving.apis 
import predict_pb2from tensorflow_serving.apis 
import prediction_service_pb2_grpc

grpc_port = 9000 # Tried also with other ports such as 8500
request = predict_pb2.PredictRequest()
request.model_spec.name = 'model'

request.model_spec.signature_name = 'serving_default'
request.inputs['input_tensor'].CopyFrom(make_tensor_proto(instance))
    options = [
        ('grpc.enable_http_proxy', 0),
        ('grpc.max_send_message_length', MAX_GRPC_MESSAGE_LENGTH),
        ('grpc.max_receive_message_length', MAX_GRPC_MESSAGE_LENGTH)
    ]

channel = grpc.insecure_channel(f'0.0.0.0:{grpc_port}', options=options)
stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)

result_future = stub.Predict.future(request, 30)  

output_tensor_proto = result_future.result().outputs['predictions']
output_shape = [dim.size for dim in output_tensor_proto.tensor_shape.dim]

output_np = np.array(output_tensor_proto.float_val).reshape(output_shape)

prediction_json = {'predictions': output_np.tolist()}

查看运行TFS的Sagemaker的docker容器,我在日志中看到rest端点被导出/公开,但gRPC端点没有被导出/公开,尽管它似乎正在运行:

ensorflow_serving/model_servers/server.cc:417] Running gRPC ModelServer at 0.0.0.0:9000 ...

与gRPC不同,在容器日志中,我可以看到rest端点已导出:

tensorflow_serving/model_servers/server.cc:438] Exporting HTTP/REST API at:localhost:8501 ...

Sagemaker TFS容器甚至支持gRPC吗?如何使用Sagemaker进行gRPC TFS预测请求?

w46czmvw

w46czmvw1#

SageMaker端点是REST端点。但是,您可以在容器内建立gRPC连接。您不能通过gRPC进行InvokeEndpoint API调用。
如果您使用的是SageMaker TensorFlow容器,则需要传递一个inference.py脚本,该脚本包含向TFS发出gRPC请求的逻辑。
请参见此示例inference.py脚本,该脚本针对TensorFlow Serving进行gRPC预测。

相关问题