Describe the usage question you have. Please include as many useful details as possible.
I have implemented my application using Apache arrow flight in golang, In the server side I created arrow record and send to the client side :
this is my DoGet function :
func (s *Server) DoGet(ticket *flight.Ticket, server flight.FlightService_DoGetServer) error {
queryResult, err := queryDatasource(dataSource, query)
if err != nil {
return err
}
defer queryResult.Close()
pool := memory.NewGoAllocator()
// ArrowSchema Parse schema to get column types and column names and returns *arrow.Schema
schema, err := queryResult.ArrowSchema()
if err != nil {
return err
}
b := array.NewRecordBuilder(pool, schema)
defer b.Release()
//WriteValues writes the values inside the *array.RecordBuilder
err = queryResult.WriteValues(b)
if err != nil {
return err
}
rec := b.NewRecord()
if err != nil{
return err
}
defer rec.Release()
rw := flight.NewRecordWriter(server, ipc.WithSchema(rec.Schema()))
defer rw.Close()
if err := rw.Write(rec); err != nil {
return err
}
return nil
}
and in the client side :
//Retrieve data
doGetClient, err := client.DoGet(ctx, doGetTicket)
if err != nil {
return fmt.Errorf("failed to get data: %v", err)
}
reader, err := flight.NewRecordReader(doGetClient, nil)
if err != nil {
fmt.Printf("Creating record reader err: %v", err)
return err
}
defer reader.Release()
var record arrow.Record
for {
rec, err := reader.Read()
if err != nil {
if err == io.EOF {
break
}
return err
}
}
As you know as the data transferring is limited by 4Mb in grpc , and I faced the error :
"received message larger than max (5000000 vs 4194304)"
how can I create chunks of data and send it to the client side and in client side how can I put back the received chunks together and use it?
Component(s)
Go
Describe the usage question you have. Please include as many useful details as possible.
I have implemented my application using Apache arrow flight in golang, In the server side I created arrow record and send to the client side :
this is my DoGet function :
and in the client side :
}
As you know as the data transferring is limited by 4Mb in grpc , and I faced the error :
"received message larger than max (5000000 vs 4194304)"
how can I create chunks of data and send it to the client side and in client side how can I put back the received chunks together and use it?
Component(s)
Go