<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/redis.git/src/rdb.c, branch psync2</title>
<subtitle>github.com: antirez/redis.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/'/>
<entry>
<title>PSYNC2: Save replication ID/offset on RDB file.</title>
<updated>2016-11-10T11:35:29+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-11-10T11:35:29+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=28c96d73b2e157a37465560bc421280d17005708'/>
<id>28c96d73b2e157a37465560bc421280d17005708</id>
<content type='text'>
This means that stopping a slave and restarting it will still make it
able to PSYNC with the master. Moreover the master itself will retain
its ID/offset, in case it gets turned into a slave, or if a slave will
try to PSYNC with it with an exactly updated offset (otherwise there is
no backlog).

This change was possible thanks to PSYNC v2 that makes saving the current
replication state much simpler.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This means that stopping a slave and restarting it will still make it
able to PSYNC with the master. Moreover the master itself will retain
its ID/offset, in case it gets turned into a slave, or if a slave will
try to PSYNC with it with an exactly updated offset (otherwise there is
no backlog).

This change was possible thanks to PSYNC v2 that makes saving the current
replication state much simpler.
</pre>
</div>
</content>
</entry>
<entry>
<title>PSYNC2: different improvements to Redis replication.</title>
<updated>2016-11-09T14:37:15+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-11-09T10:31:06+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=2669fb8364c4c4080b7b75809ca94fc8022151de'/>
<id>2669fb8364c4c4080b7b75809ca94fc8022151de</id>
<content type='text'>
The gist of the changes is that now, partial resynchronizations between
slaves and masters (without the need of a full resync with RDB transfer
and so forth), work in a number of cases when it was impossible
in the past. For instance:

1. When a slave is promoted to mastrer, the slaves of the old master can
partially resynchronize with the new master.

2. Chained slalves (slaves of slaves) can be moved to replicate to other
slaves or the master itsef, without requiring a full resync.

3. The master itself, after being turned into a slave, is able to
partially resynchronize with the new master, when it joins replication
again.

In order to obtain this, the following main changes were operated:

* Slaves also take a replication backlog, not just masters.

* Same stream replication for all the slaves and sub slaves. The
replication stream is identical from the top level master to its slaves
and is also the same from the slaves to their sub-slaves and so forth.
This means that if a slave is later promoted to master, it has the
same replication backlong, and can partially resynchronize with its
slaves (that were previously slaves of the old master).

* A given replication history is no longer identified by the `runid` of
a Redis node. There is instead a `replication ID` which changes every
time the instance has a new history no longer coherent with the past
one. So, for example, slaves publish the same replication history of
their master, however when they are turned into masters, they publish
a new replication ID, but still remember the old ID, so that they are
able to partially resynchronize with slaves of the old master (up to a
given offset).

* The replication protocol was slightly modified so that a new extended
+CONTINUE reply from the master is able to inform the slave of a
replication ID change.

* REPLCONF CAPA is used in order to notify masters that a slave is able
to understand the new +CONTINUE reply.

* The RDB file was extended with an auxiliary field that is able to
select a given DB after loading in the slave, so that the slave can
continue receiving the replication stream from the point it was
disconnected without requiring the master to insert "SELECT" statements.
This is useful in order to guarantee the "same stream" property, because
the slave must be able to accumulate an identical backlog.

* Slave pings to sub-slaves are now sent in a special form, when the
top-level master is disconnected, in order to don't interfer with the
replication stream. We just use out of band "\n" bytes as in other parts
of the Redis protocol.

An old design document is available here:

https://gist.github.com/antirez/ae068f95c0d084891305

However the implementation is not identical to the description because
during the work to implement it, different changes were needed in order
to make things working well.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The gist of the changes is that now, partial resynchronizations between
slaves and masters (without the need of a full resync with RDB transfer
and so forth), work in a number of cases when it was impossible
in the past. For instance:

1. When a slave is promoted to mastrer, the slaves of the old master can
partially resynchronize with the new master.

2. Chained slalves (slaves of slaves) can be moved to replicate to other
slaves or the master itsef, without requiring a full resync.

3. The master itself, after being turned into a slave, is able to
partially resynchronize with the new master, when it joins replication
again.

In order to obtain this, the following main changes were operated:

* Slaves also take a replication backlog, not just masters.

* Same stream replication for all the slaves and sub slaves. The
replication stream is identical from the top level master to its slaves
and is also the same from the slaves to their sub-slaves and so forth.
This means that if a slave is later promoted to master, it has the
same replication backlong, and can partially resynchronize with its
slaves (that were previously slaves of the old master).

* A given replication history is no longer identified by the `runid` of
a Redis node. There is instead a `replication ID` which changes every
time the instance has a new history no longer coherent with the past
one. So, for example, slaves publish the same replication history of
their master, however when they are turned into masters, they publish
a new replication ID, but still remember the old ID, so that they are
able to partially resynchronize with slaves of the old master (up to a
given offset).

* The replication protocol was slightly modified so that a new extended
+CONTINUE reply from the master is able to inform the slave of a
replication ID change.

* REPLCONF CAPA is used in order to notify masters that a slave is able
to understand the new +CONTINUE reply.

* The RDB file was extended with an auxiliary field that is able to
select a given DB after loading in the slave, so that the slave can
continue receiving the replication stream from the point it was
disconnected without requiring the master to insert "SELECT" statements.
This is useful in order to guarantee the "same stream" property, because
the slave must be able to accumulate an identical backlog.

* Slave pings to sub-slaves are now sent in a special form, when the
top-level master is disconnected, in order to don't interfer with the
replication stream. We just use out of band "\n" bytes as in other parts
of the Redis protocol.

An old design document is available here:

https://gist.github.com/antirez/ae068f95c0d084891305

However the implementation is not identical to the description because
during the work to implement it, different changes were needed in order
to make things working well.
</pre>
</div>
</content>
</entry>
<entry>
<title>Module: Ability to get context from IO context.</title>
<updated>2016-10-06T15:09:26+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-10-06T15:05:38+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=152c1b6802b88fbad0113adb97500d92a84e3c63'/>
<id>152c1b6802b88fbad0113adb97500d92a84e3c63</id>
<content type='text'>
It was noted by @dvirsky that it is not possible to use string functions
when writing the AOF file. This sometimes is critical since the command
rewriting may need to be built in the context of the AOF callback, and
without access to the context, and the limited types that the AOF
production functions will accept, this can be an issue.

Moreover there are other needs that we can't anticipate regarding the
ability to use Redis Modules APIs using the context in order to build
representations to emit AOF / RDB.

Because of this a new API was added that allows the user to get a
temporary context from the IO context. The context is auto released
if obtained when the RDB / AOF callback returns.

Calling multiple time the function to get the context, always returns
the same one, since it is invalid to have more than a single context.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
It was noted by @dvirsky that it is not possible to use string functions
when writing the AOF file. This sometimes is critical since the command
rewriting may need to be built in the context of the AOF callback, and
without access to the context, and the limited types that the AOF
production functions will accept, this can be an issue.

Moreover there are other needs that we can't anticipate regarding the
ability to use Redis Modules APIs using the context in order to build
representations to emit AOF / RDB.

Because of this a new API was added that allows the user to get a
temporary context from the IO context. The context is auto released
if obtained when the RDB / AOF callback returns.

Calling multiple time the function to get the context, always returns
the same one, since it is invalid to have more than a single context.
</pre>
</div>
</content>
</entry>
<entry>
<title>Modules: API to save/load single precision floating point numbers.</title>
<updated>2016-10-02T22:08:35+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-10-02T22:08:35+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=3dc84c530063e2ebd4f975e4066dd834cd6cbe9f'/>
<id>3dc84c530063e2ebd4f975e4066dd834cd6cbe9f</id>
<content type='text'>
When double precision is not needed, to take 2x space in the
serialization is not good.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When double precision is not needed, to take 2x space in the
serialization is not good.
</pre>
</div>
</content>
</entry>
<entry>
<title>Child -&gt; Parent pipe for COW info transferring.</title>
<updated>2016-09-19T11:45:20+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-09-19T11:45:20+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=e565632e599bd1801abca2359453feac51312b28'/>
<id>e565632e599bd1801abca2359453feac51312b28</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>zmalloc: zmalloc_get_smap_bytes_by_field() modified to work for any PID.</title>
<updated>2016-09-19T08:28:42+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-09-19T08:28:05+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=945a2f948eb518090325b0ceab92588a905e0f92'/>
<id>945a2f948eb518090325b0ceab92588a905e0f92</id>
<content type='text'>
The goal is to get copy-on-write amount of the child from the parent.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The goal is to get copy-on-write amount of the child from the parent.
</pre>
</div>
</content>
</entry>
<entry>
<title>Merge branch 'aofrdb' into unstable</title>
<updated>2016-09-09T13:03:21+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-09-09T13:03:21+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=3793afa0ba4a214c1e25ed74309d0594e7a1490c'/>
<id>3793afa0ba4a214c1e25ed74309d0594e7a1490c</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Fix rdb.c var types when calling rdbLoadLen().</title>
<updated>2016-09-01T09:08:44+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-09-01T09:08:44+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=57a0db94956441ac14a252cd09daa45e3f3a9453'/>
<id>57a0db94956441ac14a252cd09daa45e3f3a9453</id>
<content type='text'>
Technically as soon as Redis 64 bit gets proper support for loading
collections and/or DBs with more than 2^32 elements, the 32 bit version
should be modified in order to check if what we read from rdbLoadLen()
overflows. This would only apply to huge RDB files created with a 64 bit
instance and later loaded into a 32 bit instance.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Technically as soon as Redis 64 bit gets proper support for loading
collections and/or DBs with more than 2^32 elements, the 32 bit version
should be modified in order to check if what we read from rdbLoadLen()
overflows. This would only apply to huge RDB files created with a 64 bit
instance and later loaded into a 32 bit instance.
</pre>
</div>
</content>
</entry>
<entry>
<title>RDB AOF preamble: WIP 3 (RDB loading refactoring).</title>
<updated>2016-08-11T13:27:29+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-08-11T13:27:23+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=f1c32f0dcbe850ccc057ee1fd834c7b4f3eb0aff'/>
<id>f1c32f0dcbe850ccc057ee1fd834c7b4f3eb0aff</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>RDB AOF preamble: WIP 2.</title>
<updated>2016-08-09T14:41:40+00:00</updated>
<author>
<name>antirez</name>
<email>antirez@gmail.com</email>
</author>
<published>2016-08-09T14:41:40+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/redis.git/commit/?id=feda52381de20a6ac8ec993d20fdbaca863c0a75'/>
<id>feda52381de20a6ac8ec993d20fdbaca863c0a75</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
</feed>
