diff options
Diffstat (limited to 'email')
-rw-r--r-- | 149 |
1 files changed, 149 insertions, 0 deletions
diff --git a/email b/email new file mode 100644 index 00000000000..d09fa3c4440 --- /dev/null +++ b/email @@ -0,0 +1,149 @@ +Hi Andrei , Sujatha + +I want to discuss two things first the actual implementation and the second thing will be +the interface of circular queue with actual events processing and relay log. +So in the circular buffer granularity of reading and writing will be either one event or +one transaction.To achieve this will need to have transaction length in GTID event +itself. +>>Opinion needed:- How to store transaction length in the gtid_log_event, There are two +options +1. Store it in constant 8byte length.Issue with this is it will be 8bytes even if size +is 1 byte so lots of wasted storage. +2. Use variable memory_length to store the transaction length(net_store_length) mysql +does the same Although there are some issues with this approach + 1. get_data_size() will be accurate only after gtid_log_event::write_event is called. +circular queue member variables. + 2. We have already exhausted flags2(1 byte) in gtid_log_event so in future if we need to +store flags we have to allocate space after storing transaction length. Since length is +not fixed calculating flag will be more complex. +Athough 2nd issue can be averted by storing one(or maybe 2) byte for flag in +gtid_event_length. +|--Old Gtid_event --|1 byte flag| 4byte thread id| variable transaction size| + +Circular queue Implementation + +Members: +uchar* buffer; +uchar* buffer_end; +uint64 size; +uint elements; +/* + Some time we can have empty space in end if transaction/event is big to fit + in continues space. +*/ +uchar* buffer_usable_ptr; +uchar* write_head; +uchar* read_head; +uchar* flush_head; +std::mutex read_lock; +std::mutex write_lock; +uchar* buffer_end; +uint64 size; +uint elements; + +so as we can see there are there are total three type of pointers pointers for the +for the queue read write operation. + +write head :- this pointer in the buffer will be pointing to the current write +head all the new inserts will be done after the write head. we will be guaranteeing +that the granularity is satisfied. and transaction / events will be written in +continuous memory so in a case of when we have write head close to buffer end and +we cannot have any space for whole transaction or event then we will write from +starting of buffer given if it is empty in starting. + +read head:- reading operation will be simpler. user can read in 2 granularity +either one whole transaction or one event. reading and writing granularity should be same. + +flush head:- this will be the pointer which will be actually freeing the space +on buffer. +Some Important buffer method Implementation:- + +Empty space for this case +S= Buffer_start +F= Flush Pointer +W= write pointer +E= buffer end +U= Buffer usable_ptr + +if this case +S--F----W-----E +E - W + F -S + +if this case +S--W-- F --- U--E +F-W + +Write code +TS = Transaction size + +if this case +S--F----W----E +if W > F + if E-W > TS + then write data and W+= TS + + if E-W < TS + buffer_usable_ptr= write_head + write_head= 0 + It will become the second case + + +if this case +S--W----F--U--E +if W < F + if TS < F -W + write + W+= TS + else + give error that buffer is full + and write into file + +Read from queue + +lock the mutex +return_addr= R +if R < W + S--R---W--E + R+= TS/ES + unlock the mutex + return return_addr + +if R > W + S--W---R--U--E + R += TS/ES + if R == U + R= 0 + unlock the mutex + return return_addr +(in rest of the cases) +return NULL + +>>Opiniion needed:- I still have to figure out how I will be able to get a flush pointer. +for example let's assume we have a one binlog group commit with 10 transactions. +and our granularity is transaction level. and for simplicity let's assume we have +10 worker threads so each thread will execute one transaction and it will increment +read pointer to next transaction. so after 10 worker committing the 10 transaction +, circular queue read pointer is advanced with 10 transaction, but now the issue +is how we advance the flush pointer ? how we call back to queue that please advance +flush pointer. + +>>Opinion needed:- +second issue which I am facing is this how to differentiate whether circular buffer +is full or empty when flush point and buffer pointer point to same location on buffer. +one way is using a counter of elements second way is never have flush pointer And write +pointer pointing to same location we can have something like this that flush. pointer +is is always one less than right pointer. using elements approach is more intuitive but +it will be hard to maintain number of elements when we are not sure about how to have +flush logic. + +>>Opinion needed:- +Handling in case of full buffer:- so when circular queue is full , we will be writing +the events from io thread to temp file. +so after this we have two options +first option we can wait until buffer is empty and then copy from the temp file to buffer. +this will be done by io thread. we will be temporary stop the thread from reading the events +from network instead it will be copying events from file to buffer. +second option which is also easier to implement when we have half buffer empty we will +send a broadcast condition that buffer is half empty so the IO thread which will be +listening to this condition will copy the events from temporary file to the circular +buffer. |