Bug report #18420
Excessive, erratic memory usage
Status: | Closed | ||
---|---|---|---|
Priority: | High | ||
Assignee: | Nyall Dawson | ||
Category: | Unknown | ||
Affected QGIS version: | 3.0.0 | Regression?: | No |
Operating System: | Easy fix?: | No | |
Pull Request or Patch supplied: | Yes | Resolution: | fixed/implemented |
Crashes QGIS or corrupts data: | Yes | Copied to github as #: | 26309 |
Description
QGIS uses excessive amounts of memory, seemingly leaking some until it takes down my whole system. This seems to depend on adding and removing layers. In my case I noticed it with GeoJSON and Shapefile formats, I have no idea if it affects others).
Dataset used: https://transfer.sh/KghNi/b%C3%A4ume.7z (CC0 from https://opendata.bonn.de/dataset/baumstandorte)
There is a geojson and a shp inside.
QGIS takes about 320M of RAM with a clean (no project loaded, but some plugins) start for me.
1670M after loading the geojson after a clean start 2500M after adding the shp to that 3300M after removing the shp layer 4160M after adding the shp again 5000M after removing the shp layer again (5000M after removing the geojson layer as well but that seems to get at least partially reclaimed as adding both the shp and the geojson again only made it grow to 5075M...)
365M after loading the shp after a clean start 1720M after adding the geojson to that 1700M after removing the geojson layer 1760M after adding the geojson again
So this seemed fine (although 1.7G for a 25M GeoJSON is crazy).
365M after loading the shp after a clean start 1720M after adding the geojson to that 2530M after removing the shp layer 3350M after adding the shp again
I did not bother going further, it would have taken more memory again.
So, loading and unloading the shp file while the geojson is loaded makes QGIS gradually eat all RAM and crash the system.
QGIS version 3.0.0-Girona
QGIS code branch Release 3.0
On Archlinux
Associated revisions
[ogr] Fix ref/unref mismatch when loading OGR layers
Causes an extra connection reference which is never removed,
blocking ogr dataset closing.
Fixes #18420, probably others
[ogr] Fix ref/unref mismatch when loading OGR layers
Causes an extra connection reference which is never removed,
blocking ogr dataset closing.
Fixes #18420, probably others
(cherry-picked from f3b5838)
History
#1 Updated by Nyall Dawson almost 7 years ago
- Status changed from Open to In Progress
- Pull Request or Patch supplied changed from No to Yes
Can you test https://github.com/qgis/QGIS/pull/6593 and see if it helps?
#2 Updated by Johannes Kroeger almost 7 years ago
I could not test on the same machine yet. On my other one (same 64 bit Archlinux but different GDAL) I could not replicate the high memory usage even in an older qgis (r44842.6ed078c889) so I am a bit confused. Maybe the GDAL version plays a role too? I will try to test it tomorrow.
#3 Updated by Nyall Dawson almost 7 years ago
Gdal 2.3 will help a lot here (it's gdal which consumes all the memory on opening the geojson)
#4 Updated by Johannes Kroeger almost 7 years ago
Yeah, that one had the latest GDAL from SVN I think.
On the machine where I noticed the problem:
QGIS 3.1.0-Master (193c554b4c) with the patch applied.
GDAL 2.2.3, released 2017/11/20
1660M added GeoJSON
2033M added shp
3000M removed shp
3800M added shp
4200M removed shp
It also does not free after unloading all layers or creating a new project.
#5 Updated by Nyall Dawson almost 7 years ago
- % Done changed from 0 to 100
- Status changed from In Progress to Closed
Applied in changeset qgis|f3b5838f5e1d65ab17bef6ff0480daf6c89f5424.
#6 Updated by Johannes Kroeger almost 7 years ago
- Status changed from Closed to Reopened
- Assignee set to Nyall Dawson
No luck :(
I built from Git r46587.10044fb1dd with the same GDAL 2.2.3, released 2017/11/20 and tried the files again:
1660 gj
2500 shp+
3300 shp-
3300 shp+
3300 shp-
4200 shp+
5000 shp-
5000 shp+
5000 shp-
5000 shp+
5000 shp-
5000 gj-
5000 new project
5000 shp+
5100 gj+
5100 shp-
5100 shp+
5100 open random other project
#7 Updated by Giovanni Manghi about 6 years ago
- Status changed from Reopened to Feedback
Please try on QGIS 3.4.1, if the issue is still valid change the affected version, thanks.
#8 Updated by Johannes Kroeger about 6 years ago
- Status changed from Feedback to Open
Tested in 3.2.3-Bonn (Compiled against GDAL/OGR 2.3.1, Running against GDAL/OGR 2.3.2).
I could not reproduce the excessive memory usage. So this issue can be closed.
#9 Updated by Giovanni Manghi about 6 years ago
- Status changed from Open to Closed
- Resolution set to fixed/implemented
You should test 3.4.1 as 3.2 is not supported anymore, so reopen if necessary.