<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/haproxy.git, branch 20211020-res-fix-6</title>
<subtitle>github.com: haproxy/haproxy.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/'/>
<entry>
<title>MEDIUM: resolvers: remove the last occurrences of the "safe" argument</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-20T12:07:31+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=46968e1f69178950eb1a0862d9e5996f260a4fae'/>
<id>46968e1f69178950eb1a0862d9e5996f260a4fae</id>
<content type='text'>
This one was used to indicate whether the callee had to follow particularly
safe code path when removing resolutions. Since the code now uses a kill
list, this is not needed anymore.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This one was used to indicate whether the callee had to follow particularly
safe code path when removing resolutions. Since the code now uses a kill
list, this is not needed anymore.
</pre>
</div>
</content>
</entry>
<entry>
<title>MEDIUM: resolvers: use a kill list to preserve the list consistency</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-18T14:46:38+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=d8b7353033e840688fa445c5c24e3f364bbbaf92'/>
<id>d8b7353033e840688fa445c5c24e3f364bbbaf92</id>
<content type='text'>
When scanning resolution.curr it's possible to try to free some
resolutions which will themselves result in freeing other ones. If
one of these other ones is exactly the next one in the list, the list
walk visits deleted nodes and causes memory corruption, double-frees
and so on. The approach taken using the "safe" argument to some
functions seems to work but it's extremely brittle as it is required
to carefully check all call paths from process_ressolvers() and pass
the argument to 1 there to refrain from deleting entries, so the bug
is very likely to come back after some tiny changes to this code.

A variant was tried, checking at various places that the current task
corresponds to process_resolvers() but this is also quite brittle even
though a bit less.

This patch uses another approach which consists in carefully unlinking
elements from the list and deferring their removal by placing it in a
kill list instead of deleting them synchronously. The real benefit here
is that the complexity only has to be placed where the complications
are.

A thread-local list is fed with elements to be deleted before scanning
the resolutions, and it's flushed at the end by picking the first one
until the list is empty. This way we never dereference the next element
and do not care about its presence or not in the list. One function,
resolv_unlink_resolution(), is exported and used outside, so it had to
be modified to use this list as well. Internal code has to use
_resolv_unlink_resolution() instead.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When scanning resolution.curr it's possible to try to free some
resolutions which will themselves result in freeing other ones. If
one of these other ones is exactly the next one in the list, the list
walk visits deleted nodes and causes memory corruption, double-frees
and so on. The approach taken using the "safe" argument to some
functions seems to work but it's extremely brittle as it is required
to carefully check all call paths from process_ressolvers() and pass
the argument to 1 there to refrain from deleting entries, so the bug
is very likely to come back after some tiny changes to this code.

A variant was tried, checking at various places that the current task
corresponds to process_resolvers() but this is also quite brittle even
though a bit less.

This patch uses another approach which consists in carefully unlinking
elements from the list and deferring their removal by placing it in a
kill list instead of deleting them synchronously. The real benefit here
is that the complexity only has to be placed where the complications
are.

A thread-local list is fed with elements to be deleted before scanning
the resolutions, and it's flushed at the end by picking the first one
until the list is empty. This way we never dereference the next element
and do not care about its presence or not in the list. One function,
resolv_unlink_resolution(), is exported and used outside, so it had to
be modified to use this list as well. Internal code has to use
_resolv_unlink_resolution() instead.
</pre>
</div>
</content>
</entry>
<entry>
<title>CLEANUP: resolvers: replace all LIST_DELETE with LIST_DEL_INIT</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-19T20:01:36+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=6fb5b6d5c7a1cacdb345b185fab205a0ff685e0f'/>
<id>6fb5b6d5c7a1cacdb345b185fab205a0ff685e0f</id>
<content type='text'>
The code as it is uses crossed lists between many elements, and at
many places the code relies on list iterators or emptiness checks,
which does not work with only LIST_DELETE. Further, it is quite
difficult to place debugging code and checks in the current situation,
and gdb is helpless.

This code replaces all LIST_DELETE calls with LIST_DEL_INIT so that
it becomes possible to trust the lists.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The code as it is uses crossed lists between many elements, and at
many places the code relies on list iterators or emptiness checks,
which does not work with only LIST_DELETE. Further, it is quite
difficult to place debugging code and checks in the current situation,
and gdb is helpless.

This code replaces all LIST_DELETE calls with LIST_DEL_INIT so that
it becomes possible to trust the lists.
</pre>
</div>
</content>
</entry>
<entry>
<title>CLEANUP: resolvers: simplify resolv_link_resolution() regarding requesters</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-19T09:59:25+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=2b3d4b6214dd998e795d0a88cb601655a3b3b2e6'/>
<id>2b3d4b6214dd998e795d0a88cb601655a3b3b2e6</id>
<content type='text'>
This function allocates requesters by hand for each and every type. This
is complex and error-prone, and it doesn't even initialize the list part,
leaving dangling pointers that complicate debugging.

This patch introduces a new function resolv_get_requester() that either
returns the current pointer if valid or tries to allocate a new one and
links it to its destination. Then it makes use of it in the function
above to clean it up quite a bit. This allows to remove complicated but
unneeded tests.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This function allocates requesters by hand for each and every type. This
is complex and error-prone, and it doesn't even initialize the list part,
leaving dangling pointers that complicate debugging.

This patch introduces a new function resolv_get_requester() that either
returns the current pointer if valid or tries to allocate a new one and
links it to its destination. Then it makes use of it in the function
above to clean it up quite a bit. This allows to remove complicated but
unneeded tests.
</pre>
</div>
</content>
</entry>
<entry>
<title>CLEANUP: always initialize the answer_list</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-19T09:29:21+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=c1ab34d2cf8049c95ece6d5bef20b6e967a6d235'/>
<id>c1ab34d2cf8049c95ece6d5bef20b6e967a6d235</id>
<content type='text'>
Similar to the previous patch, the answer's list was only initialized the
first time it was added to a list, leading to bogus outdated pointer to
appear when debugging code is added around it to watch it. Let's make
sure it's always initialized upon allocation.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Similar to the previous patch, the answer's list was only initialized the
first time it was added to a list, leading to bogus outdated pointer to
appear when debugging code is added around it to watch it. Let's make
sure it's always initialized upon allocation.
</pre>
</div>
</content>
</entry>
<entry>
<title>BUG/MEDIUM: resolvers: always check a valid item in query_list</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-19T09:17:33+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=c90363c9ad40341a8c9653b4e8ab6a6a1ced2224'/>
<id>c90363c9ad40341a8c9653b4e8ab6a6a1ced2224</id>
<content type='text'>
The query_list is physically stored in the struct resolution itself,
so we have a list that contains a list to items stored in itself (and
there is a single item). But the list is first initialized in
resolv_validate_dns_response(), while it's scanned in
resolv_process_responses() later after calling the former. First,
this results in crashes as soon as the code is instrumented a little
bit for debugging, as elements from a previous incarnation can appear.

But in addition to this, the presence of an element is checked by
verifying that the return of LIST_NEXT() is not NULL, while it may
never be NULL even for an empty list, resulting in bugs or crashes
if the number of responses does not match the list's contents. This
is easily triggered by testing for the list non-emptiness outside of
the function.

Let's make sure the list is always correct, i.e. it's initialized to
an empty list when the structure is allocated, elements are checked by
first verifying the list is not empty, they are deleted once checked,
and in any case at end so that there are no dangling pointers.

This should be backported, but only as long as the patch fits without
modifications, as adaptations can be risky there given that bugs tend
to hide each other.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The query_list is physically stored in the struct resolution itself,
so we have a list that contains a list to items stored in itself (and
there is a single item). But the list is first initialized in
resolv_validate_dns_response(), while it's scanned in
resolv_process_responses() later after calling the former. First,
this results in crashes as soon as the code is instrumented a little
bit for debugging, as elements from a previous incarnation can appear.

But in addition to this, the presence of an element is checked by
verifying that the return of LIST_NEXT() is not NULL, while it may
never be NULL even for an empty list, resulting in bugs or crashes
if the number of responses does not match the list's contents. This
is easily triggered by testing for the list non-emptiness outside of
the function.

Let's make sure the list is always correct, i.e. it's initialized to
an empty list when the structure is allocated, elements are checked by
first verifying the list is not empty, they are deleted once checked,
and in any case at end so that there are no dangling pointers.

This should be backported, but only as long as the patch fits without
modifications, as adaptations can be risky there given that bugs tend
to hide each other.
</pre>
</div>
</content>
</entry>
<entry>
<title>CLEANUP: resolvers: do not export resolv_purge_resolution_answer_records()</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-19T09:16:11+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=90bd6ffb3b8e94e4205b8d22c9a6c2cc9c702734'/>
<id>90bd6ffb3b8e94e4205b8d22c9a6c2cc9c702734</id>
<content type='text'>
This code is dangerous enough that we certainly don't want external code
to ever approach it, let's not export unnecessary functions like this one.
It was made static and a comment was added about its purpose.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
This code is dangerous enough that we certainly don't want external code
to ever approach it, let's not export unnecessary functions like this one.
It was made static and a comment was added about its purpose.
</pre>
</div>
</content>
</entry>
<entry>
<title>BUG/MAJOR: resolvers: add other missing references during resolution removal</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-18T14:49:14+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=b34ef16a34e5ff2bab4e9b5f9013c33141d9614b'/>
<id>b34ef16a34e5ff2bab4e9b5f9013c33141d9614b</id>
<content type='text'>
There is a fundamental design bug in the resolvers code which is that
a list of active resolutions is being walked to try to delete outdated
entries, and that the code responsible for removing them also removes
other elements, including the next one which will be visited by the
list iterator. This randomly causes a use-after-free condition leading
to crashes, infinite loops and various other issues such as random memory
corruption.

A first fix for the memory fix for this was brought by commit 0efc0993e
("BUG/MEDIUM: resolvers: Don't release resolution from a requester
callbacks"). While preparing for more fixes, some code was factored by
commit 11c6c3965 ("MINOR: resolvers: Clean server in a dedicated function
when removing a SRV item"), which inadvertently passed "0" as the "safe"
argument all the time, missing one case of removal protection, instead
of always using "safe". This patch reintroduces the correct argument.

This must be backported with all fixes above.

Cc: Christopher Faulet &lt;cfaulet@haproxy.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
There is a fundamental design bug in the resolvers code which is that
a list of active resolutions is being walked to try to delete outdated
entries, and that the code responsible for removing them also removes
other elements, including the next one which will be visited by the
list iterator. This randomly causes a use-after-free condition leading
to crashes, infinite loops and various other issues such as random memory
corruption.

A first fix for the memory fix for this was brought by commit 0efc0993e
("BUG/MEDIUM: resolvers: Don't release resolution from a requester
callbacks"). While preparing for more fixes, some code was factored by
commit 11c6c3965 ("MINOR: resolvers: Clean server in a dedicated function
when removing a SRV item"), which inadvertently passed "0" as the "safe"
argument all the time, missing one case of removal protection, instead
of always using "safe". This patch reintroduces the correct argument.

This must be backported with all fixes above.

Cc: Christopher Faulet &lt;cfaulet@haproxy.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>DEBUG: dns: add a few more BUG_ON at sensitive places</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-20T09:02:13+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=e6386f87fd414fc4cc6a2202afae7a4589840c02'/>
<id>e6386f87fd414fc4cc6a2202afae7a4589840c02</id>
<content type='text'>
A few places have been caught triggering late bugs recently, always cases
of use-after-free because a freed element was still found in one of the
lists. This patch adds a few checks for such elements in dns_session_free()
before the final pool_free() and dns_session_io_handler() before adding
elements to lists to make sure they remain consistent. They do not trigger
anymore now.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
A few places have been caught triggering late bugs recently, always cases
of use-after-free because a freed element was still found in one of the
lists. This patch adds a few checks for such elements in dns_session_free()
before the final pool_free() and dns_session_io_handler() before adding
elements to lists to make sure they remain consistent. They do not trigger
anymore now.
</pre>
</div>
</content>
</entry>
<entry>
<title>CLEANUP: dns: always detach the appctx from the dns session on release</title>
<updated>2021-10-20T13:14:13+00:00</updated>
<author>
<name>Willy Tarreau</name>
<email>w@1wt.eu</email>
</author>
<published>2021-10-20T12:38:43+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/haproxy.git/commit/?id=155171d1fa6782157891716a910436efffb7848b'/>
<id>155171d1fa6782157891716a910436efffb7848b</id>
<content type='text'>
When dns_session_release() calls dns_session_free(), it was shown that
it might still be attached there:

  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  0x00000000006437d7 in dns_session_free (ds=0x7f895439e810) at src/dns.c:768
  768             BUG_ON(!LIST_ISEMPTY(&amp;ds-&gt;ring.waiters));
  [Current thread is 1 (Thread 0x7f895bbe2700 (LWP 31792))]
  (gdb) bt
  #0  0x00000000006437d7 in dns_session_free (ds=0x7f895439e810) at src/dns.c:768
  #1  0x0000000000643ab8 in dns_session_release (appctx=0x7f89545a4ff0) at src/dns.c:805
  #2  0x000000000062e35a in si_applet_release (si=0x7f89545a5550) at include/haproxy/stream_interface.h:236
  #3  0x000000000063150f in stream_int_shutw_applet (si=0x7f89545a5550) at src/stream_interface.c:1697
  #4  0x0000000000640ab8 in si_shutw (si=0x7f89545a5550) at include/haproxy/stream_interface.h:437
  #5  0x0000000000643103 in dns_session_io_handler (appctx=0x7f89545a4ff0) at src/dns.c:725
  #6  0x00000000006d776f in task_run_applet (t=0x7f89545a5100, context=0x7f89545a4ff0, state=81924) at src/applet.c:90
  #7  0x000000000068b82b in run_tasks_from_lists (budgets=0x7f895bbbf5c0) at src/task.c:611
  #8  0x000000000068c258 in process_runnable_tasks () at src/task.c:850
  #9  0x0000000000621e61 in run_poll_loop () at src/haproxy.c:2636
  #10 0x0000000000622328 in run_thread_poll_loop (data=0x8d7440 &lt;ha_thread_info+64&gt;) at src/haproxy.c:2807
  #11 0x00007f895c54a06b in start_thread () from /lib64/libpthread.so.0
  #12 0x00007f895bf3772f in clone () from /lib64/libc.so.6
  (gdb) p &amp;ds-&gt;ring.waiters
  $1 = (struct list *) 0x7f895439e8a8
  (gdb) p ds-&gt;ring.waiters
  $2 = {
    n = 0x7f89545a5078,
    p = 0x7f89545a5078
  }
  (gdb) p ds-&gt;ring.waiters-&gt;n
  $3 = (struct list *) 0x7f89545a5078
  (gdb) p *ds-&gt;ring.waiters-&gt;n
  $4 = {
    n = 0x7f895439e8a8,
    p = 0x7f895439e8a8
  }

Let's always detach it before freeing so that it remains possible to
check the dns_session's ring before releasing it, and possibly catch
bugs.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
When dns_session_release() calls dns_session_free(), it was shown that
it might still be attached there:

  Program terminated with signal SIGSEGV, Segmentation fault.
  #0  0x00000000006437d7 in dns_session_free (ds=0x7f895439e810) at src/dns.c:768
  768             BUG_ON(!LIST_ISEMPTY(&amp;ds-&gt;ring.waiters));
  [Current thread is 1 (Thread 0x7f895bbe2700 (LWP 31792))]
  (gdb) bt
  #0  0x00000000006437d7 in dns_session_free (ds=0x7f895439e810) at src/dns.c:768
  #1  0x0000000000643ab8 in dns_session_release (appctx=0x7f89545a4ff0) at src/dns.c:805
  #2  0x000000000062e35a in si_applet_release (si=0x7f89545a5550) at include/haproxy/stream_interface.h:236
  #3  0x000000000063150f in stream_int_shutw_applet (si=0x7f89545a5550) at src/stream_interface.c:1697
  #4  0x0000000000640ab8 in si_shutw (si=0x7f89545a5550) at include/haproxy/stream_interface.h:437
  #5  0x0000000000643103 in dns_session_io_handler (appctx=0x7f89545a4ff0) at src/dns.c:725
  #6  0x00000000006d776f in task_run_applet (t=0x7f89545a5100, context=0x7f89545a4ff0, state=81924) at src/applet.c:90
  #7  0x000000000068b82b in run_tasks_from_lists (budgets=0x7f895bbbf5c0) at src/task.c:611
  #8  0x000000000068c258 in process_runnable_tasks () at src/task.c:850
  #9  0x0000000000621e61 in run_poll_loop () at src/haproxy.c:2636
  #10 0x0000000000622328 in run_thread_poll_loop (data=0x8d7440 &lt;ha_thread_info+64&gt;) at src/haproxy.c:2807
  #11 0x00007f895c54a06b in start_thread () from /lib64/libpthread.so.0
  #12 0x00007f895bf3772f in clone () from /lib64/libc.so.6
  (gdb) p &amp;ds-&gt;ring.waiters
  $1 = (struct list *) 0x7f895439e8a8
  (gdb) p ds-&gt;ring.waiters
  $2 = {
    n = 0x7f89545a5078,
    p = 0x7f89545a5078
  }
  (gdb) p ds-&gt;ring.waiters-&gt;n
  $3 = (struct list *) 0x7f89545a5078
  (gdb) p *ds-&gt;ring.waiters-&gt;n
  $4 = {
    n = 0x7f895439e8a8,
    p = 0x7f895439e8a8
  }

Let's always detach it before freeing so that it remains possible to
check the dns_session's ring before releasing it, and possibly catch
bugs.
</pre>
</div>
</content>
</entry>
</feed>
