<feed xmlns='http://www.w3.org/2005/Atom'>
<title>delta/python-packages/psutil.git/docs, branch wheel4</title>
<subtitle>github.com: giampaolo/psutil.git
</subtitle>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/'/>
<entry>
<title>Memory leak test: take fluctuations into account (#1757)</title>
<updated>2020-05-12T22:40:04+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-05-12T22:40:04+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=6adcca6c9fda5a36e923bfca3591b972e98a0ce5'/>
<id>6adcca6c9fda5a36e923bfca3591b972e98a0ce5</id>
<content type='text'>
Preamble
=======

We have a [memory leak test suite](https://github.com/giampaolo/psutil/blob/e1ea2bccf8aea404dca0f79398f36f37217c45f6/psutil/tests/__init__.py#L897), which calls a function many times and fails if the process memory increased. We do this in order to detect missing `free()` or `Py_DECREF` calls in the C modules. When we do, then we have a memory leak.

The problem
==========

A problem we've been having for probably over 10 years, is the false positives. That's because the memory fluctuates. Sometimes it may increase (or even decrease!) due to how the OS handles memory, the Python's garbage collector, the fact that RSS is an approximation and who knows what else. So thus far we tried to compensate that by using the following logic:
- warmup (call fun 10 times)
- call the function many times (1000)
- if memory increased before/after calling function 1000 times, then keep calling it for another 3 secs
- if it still increased at all (&gt; 0) then fail

This logic didn't really solve the problem, as we still had occasional false positives, especially lately on FreeBSD. 

The solution
=========

This PR changes the internal algorithm so that in case of failure (mem &gt; 0 after calling fun() N times) we retry the test for up to 5 times, increasing N (repetitions) each time, so we consider it a failure only if the memory **keeps increasing** between runs. So for instance, here's a legitimate failure:

```
psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions ... 
Run #1: extra-mem=696.0K, per-call=3.5K, calls=200
Run #2: extra-mem=1.4M, per-call=3.5K, calls=400
Run #3: extra-mem=2.1M, per-call=3.5K, calls=600
Run #4: extra-mem=2.7M, per-call=3.5K, calls=800
Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000
FAIL
```

If, on the other hand, the memory increased on one run (say 200 calls) but decreased on the next run (say 400 calls), then it clearly means it's a false positive, because memory consumption may be &gt; 0 on second run, but if it's lower than the previous run with less repetitions, then it cannot possibly represent a leak (just a fluctuation):

```
psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_net_connections ... 
Run #1: extra-mem=568.0K, per-call=2.8K, calls=200
Run #2: extra-mem=24.0K, per-call=61.4B, calls=400
OK
```

Note about mallinfo()
================

Aka #1275. `mallinfo()` on Linux is supposed to provide memory metrics about how many bytes gets allocated on the heap by `malloc()`, so it's supposed to be way more precise than RSS and also [USS](http://grodola.blogspot.com/2016/02/psutil-4-real-process-memory-and-environ.html). In another branch were I exposed it, I verified that fluctuations still occur even when using `mallinfo()` though, despite less often. So that means even `mallinfo()` would not grant 100% stability.</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Preamble
=======

We have a [memory leak test suite](https://github.com/giampaolo/psutil/blob/e1ea2bccf8aea404dca0f79398f36f37217c45f6/psutil/tests/__init__.py#L897), which calls a function many times and fails if the process memory increased. We do this in order to detect missing `free()` or `Py_DECREF` calls in the C modules. When we do, then we have a memory leak.

The problem
==========

A problem we've been having for probably over 10 years, is the false positives. That's because the memory fluctuates. Sometimes it may increase (or even decrease!) due to how the OS handles memory, the Python's garbage collector, the fact that RSS is an approximation and who knows what else. So thus far we tried to compensate that by using the following logic:
- warmup (call fun 10 times)
- call the function many times (1000)
- if memory increased before/after calling function 1000 times, then keep calling it for another 3 secs
- if it still increased at all (&gt; 0) then fail

This logic didn't really solve the problem, as we still had occasional false positives, especially lately on FreeBSD. 

The solution
=========

This PR changes the internal algorithm so that in case of failure (mem &gt; 0 after calling fun() N times) we retry the test for up to 5 times, increasing N (repetitions) each time, so we consider it a failure only if the memory **keeps increasing** between runs. So for instance, here's a legitimate failure:

```
psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_disk_partitions ... 
Run #1: extra-mem=696.0K, per-call=3.5K, calls=200
Run #2: extra-mem=1.4M, per-call=3.5K, calls=400
Run #3: extra-mem=2.1M, per-call=3.5K, calls=600
Run #4: extra-mem=2.7M, per-call=3.5K, calls=800
Run #5: extra-mem=3.4M, per-call=3.5K, calls=1000
FAIL
```

If, on the other hand, the memory increased on one run (say 200 calls) but decreased on the next run (say 400 calls), then it clearly means it's a false positive, because memory consumption may be &gt; 0 on second run, but if it's lower than the previous run with less repetitions, then it cannot possibly represent a leak (just a fluctuation):

```
psutil.tests.test_memory_leaks.TestModuleFunctionsLeaks.test_net_connections ... 
Run #1: extra-mem=568.0K, per-call=2.8K, calls=200
Run #2: extra-mem=24.0K, per-call=61.4B, calls=400
OK
```

Note about mallinfo()
================

Aka #1275. `mallinfo()` on Linux is supposed to provide memory metrics about how many bytes gets allocated on the heap by `malloc()`, so it's supposed to be way more precise than RSS and also [USS](http://grodola.blogspot.com/2016/02/psutil-4-real-process-memory-and-environ.html). In another branch were I exposed it, I verified that fluctuations still occur even when using `mallinfo()` though, despite less often. So that means even `mallinfo()` would not grant 100% stability.</pre>
</div>
</content>
</entry>
<entry>
<title>Process wait() improvements (#1747)</title>
<updated>2020-05-02T22:19:53+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-05-02T22:19:53+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=42368e6a786415d932cb48d0fd006a0c302ab31b'/>
<id>42368e6a786415d932cb48d0fd006a0c302ab31b</id>
<content type='text'>
* `Process.wait()` on POSIX now returns an `enum` showing the negative which was used to terminate the process:
```python
&gt;&gt;&gt; import psutil
&gt;&gt;&gt; p = psutil.Process(9891)
&gt;&gt;&gt; p.terminate()
&gt;&gt;&gt; p.wait()
&lt;Negsignal.SIGTERM: -15&gt;
```

* the return value is cached so that the exit code can be retrieved on then next call, mimicking `subprocess.Popen.wait()`
* `Process` object provides more `status` and `exitcode` additional info on `str()` and `repr()`:

```
 &gt;&gt;&gt; proc
 psutil.Process(pid=12739, name='python3', status='terminated', exitcode=&lt;Negsigs.SIGTERM: -15&gt;, started='15:08:20')
```

Extra:
* improved `wait()` doc
* reverted #1736: `psutil.Popen` uses original `subprocess.Popen.wait` method (safer)</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
* `Process.wait()` on POSIX now returns an `enum` showing the negative which was used to terminate the process:
```python
&gt;&gt;&gt; import psutil
&gt;&gt;&gt; p = psutil.Process(9891)
&gt;&gt;&gt; p.terminate()
&gt;&gt;&gt; p.wait()
&lt;Negsignal.SIGTERM: -15&gt;
```

* the return value is cached so that the exit code can be retrieved on then next call, mimicking `subprocess.Popen.wait()`
* `Process` object provides more `status` and `exitcode` additional info on `str()` and `repr()`:

```
 &gt;&gt;&gt; proc
 psutil.Process(pid=12739, name='python3', status='terminated', exitcode=&lt;Negsigs.SIGTERM: -15&gt;, started='15:08:20')
```

Extra:
* improved `wait()` doc
* reverted #1736: `psutil.Popen` uses original `subprocess.Popen.wait` method (safer)</pre>
</div>
</content>
</entry>
<entry>
<title>Revert #1736: Popen inheriting from subprocess (#1744)</title>
<updated>2020-05-01T09:20:52+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-05-01T09:20:52+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=8d8a7804d159e5b80378000b57bbfbaf63ce6e8f'/>
<id>8d8a7804d159e5b80378000b57bbfbaf63ce6e8f</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>psutil.Popen: inherit from subprocess + support wait(timeout=...) parameter (#1736)</title>
<updated>2020-04-28T01:32:46+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-04-28T01:32:46+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=92e150ef5e309ff93378ae4538065f1ca5c00a17'/>
<id>92e150ef5e309ff93378ae4538065f1ca5c00a17</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Parallel tests (UNIX) (#1709)</title>
<updated>2020-04-26T00:02:43+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-04-26T00:02:43+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=420c9e82fd3321e3062424846870f3104c1b0eb8'/>
<id>420c9e82fd3321e3062424846870f3104c1b0eb8</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>sensors_fans is not available on MacOS (#1710)</title>
<updated>2020-03-04T16:25:18+00:00</updated>
<author>
<name>crusaderky</name>
<email>crusaderky@gmail.com</email>
</author>
<published>2020-03-04T16:25:18+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=ea4887e758ee2845f8e238655aaadb56f504e128'/>
<id>ea4887e758ee2845f8e238655aaadb56f504e128</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Git hook for renamed/added/deleted files + flake8 print() + tidelift (#1704)</title>
<updated>2020-02-21T11:40:20+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-02-21T11:40:20+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=567547fa3ba3f11ee4f2dc9e73d37a146fe49e1b'/>
<id>567547fa3ba3f11ee4f2dc9e73d37a146fe49e1b</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>Add C linter script (#1698)</title>
<updated>2020-02-18T17:44:43+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-02-18T17:44:43+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=5e47e0bcba7585909a11e9c7424e864fc26a7fd2'/>
<id>5e47e0bcba7585909a11e9c7424e864fc26a7fd2</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>revert process_iter() exactly how it was pre #1667</title>
<updated>2020-02-18T01:39:13+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-02-18T01:39:13+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=41fcba5dd7ecbbc788814c722f31e630422ba28e'/>
<id>41fcba5dd7ecbbc788814c722f31e630422ba28e</id>
<content type='text'>
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
</pre>
</div>
</content>
</entry>
<entry>
<title>revert #1667 process_iter() new_only param</title>
<updated>2020-02-18T00:43:16+00:00</updated>
<author>
<name>Giampaolo Rodola</name>
<email>g.rodola@gmail.com</email>
</author>
<published>2020-02-18T00:43:16+00:00</published>
<link rel='alternate' type='text/html' href='http://git.baserock.org/cgit/delta/python-packages/psutil.git/commit/?id=c9fc4fdadc5a19d97916cbf13b1b479834b94bd8'/>
<id>c9fc4fdadc5a19d97916cbf13b1b479834b94bd8</id>
<content type='text'>
On a second thought I realized that process_iter() uses a global
variable, so it's not thread safe.
That means that if the are 2 threads using it, the first thread one calling the function (+ consume the iterator), will "steal" the processes of the second thread.

psutil.cpu_percent() has the same problem. That means we have a problem
can't solve with the current API and requires a lot of thinking on how
to solve it as it's not obvious.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
On a second thought I realized that process_iter() uses a global
variable, so it's not thread safe.
That means that if the are 2 threads using it, the first thread one calling the function (+ consume the iterator), will "steal" the processes of the second thread.

psutil.cpu_percent() has the same problem. That means we have a problem
can't solve with the current API and requires a lot of thinking on how
to solve it as it's not obvious.
</pre>
</div>
</content>
</entry>
</feed>
