Bug report #12578

Large pause before snapping to large layer

Added by francesco marucci over 9 years ago. Updated almost 9 years ago.

Status:Closed
Priority:High
Assignee:-
Category:Digitising
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:
Crashes QGIS or corrupts data:Yes Copied to github as #:20703

Description

New description:

steps to replicate:

1 add a layer A with a lot of vertexes (the more vertexes the more this issue will be clear)
2 add another layer B
3 enable snapping to layer A
4 start editing layer B, snap a vertex of B to A -> a certain amount of RAM starts to be used in addition to the one that qgis was using before 4)
5 snapping more B vertexes to A does not change the amount of memory used
6 add a layer C with a lot of vertexes and enable snapping to C
7 snap a vertex from B to C -> a certain amount of RAM starts to be used in addition to the one that qgis was using before 7)
8 repeat until memory finishes

Old description:

Dear all, we noticed that qGis use a lot of memory when start to snap to a big layer during an edit session.

This issue is verified when we edit a layer (with few features) and the snap to another very big layer is activated: at first edit activity (add a new vertex of a new feature or first move a vertex of an existing feature), qGis starts to load a lot of memory and meanwhile qGis is blocked until all the memory is loaded.

We discovered this issue with a very big postgis layer (more than 1 million polygons), the waiting time is about 1 minute and the RAM occupied is 800Mb.
qGis free the RAM only when the project is closed but it is occupied during all the editing session.
The occupied RAM is dependent from the number of layer in TOC
For bigger layers, or also if we have more than layer in the TOC, sometimes qGIs crashes (due to the memory limits, I guess).

How to reproduce this issue:

- load a small polygon layer (edit)
- load a very big polygon layer (snapTo)
- activate the snap on the shapTo layer (every kind of snap)
- start editing the edit layer
- move one vertex

You will see qGis blocked and a lot of memory loading (in task manager).

You can download the test dataset from:
http://mappegis.regione.emilia-romagna.it/gstatico/documenti/fra/snapping_issue.zip (105Mb)

thanks
francesco

qgis_loading_ram_snap.jpg (143 KB) francesco marucci, 2015-04-15 02:49 AM

Associated revisions

Revision fa66583e
Added by Martin Dobias almost 9 years ago

Improve indexing strategy for snapping (fixes #12578)

Implemented a simple heuristic that should keep the number of cached
features per layer reasonable - and thus lower the amount of consumed
memory and CPU for big layers.

History

#1 Updated by Giovanni Manghi over 9 years ago

  • Status changed from Open to Feedback

was this issue also present in older qgis releases? or it just a 2.8.1 thing? thanks.

#2 Updated by Giovanni Manghi over 9 years ago

I just tested your project on qgis master on Linux, and I cannot replicate this issue. When starting editing the used memory by qgis raises to around 400mb, but then it remains stable.

I see two possibilities: the issue has been fixed on master or it is just a Windows thing. I cannot do more tests right now.

#3 Updated by francesco marucci over 9 years ago

we noticed this issue only on 2.8.1, untill 2.6 all was correct.
I tested only in win7, with the osgeo4w 32&64 installer and also with the standalone installers (32&64)). i tried also with the master build 33.

i will try to reproduce the dataset i used to do the tests (bigger than i sent) and i will send you.
in any case in my win7 pc, with thae dataset i sent, before starting digitising the memory occupied by qgis was 140mb, after more than 300mb. if you have more snapped layers in toc, the memory used is the summatory of each layer.

regards
francesco

#4 Updated by francesco marucci over 9 years ago

dear all,
we grab a video showing the problem:
http://mappegis.regione.emilia-romagna.it/gstatico/documenti/fra/VID_20150417_124632.mp4

in the toc we have tree "snapped" layers with a total of 20 millions vertices...
you can see that the memory used when the first editing activity (new polygon) starts is about 1 Gb.

regards,
francesco

#5 Updated by Giovanni Manghi over 9 years ago

  • Affected QGIS version changed from 2.8.1 to master
  • Subject changed from Problem with the snapping to layer function: a lot of memory loaded during the first action to Snapping to layer function: memory use grows indefinitely
  • Status changed from Feedback to Open

#6 Updated by Matthias Kuhn over 9 years ago

  • Status changed from Open to Closed
  • Resolution set to fixed/implemented

Likely solved in 8b6abacc99

It would be appreciated if you can verify with tomorrows nightly build and reopen if the problem persists.

#7 Updated by Giovanni Manghi over 9 years ago

  • Resolution deleted (fixed/implemented)
  • Target version set to Future Release - High Priority
  • Status changed from Closed to Reopened

It would be appreciated if you can verify with tomorrows nightly build and reopen if the problem persists.

unfortunately still there.

#8 Updated by Paolo Cavallini over 9 years ago

Possibly related:
  • load a large layer
  • add a new empty layer (either shp or in memory)
    This takes an enormous amount of time.
    I confirm this makes QGIS almost unusable with very large layers.

#9 Updated by Paolo Cavallini over 9 years ago

It might be related to the creation of the index for snapping.

#10 Updated by Matt Travis over 9 years ago

I tested the release candidate yesterday and it is still the same. Do we think this will be resolved in 2.10 and the ltr update?

#11 Updated by Saber Razmjooei over 9 years ago

The problem is in 2.10 for sure. See Martin's email (2nd in the thread) regarding the new geometry engine and snapping cache:
http://osgeo-org.1560.x6.nabble.com/Potentially-serious-performance-regression-in-new-geometry-should-2-10-be-delayed-td5212408.html

#12 Updated by Saber Razmjooei about 9 years ago

  • Status changed from Reopened to Feedback

Hi Matt,
Tried it with a 0.5 GB shapefile:
https://geoportal.statistics.gov.uk/Docs/Boundaries/Output_areas_(E+W)_2011_Boundaries_(Full_Extent)_V2.zip
and it works fine with the latest master. I have deleted previous spatial index and created a new one.

Could you upload or link to some sample data?

#13 Updated by Nyall Dawson about 9 years ago

I can't reproduce with master either. There were a few (tiny) leaks flagged which I'll fix shortly, but nothing significant.

Just to confirm - when you say it's still present, are you referring to the memory leak or the initial hang when the snapping index is built?

#14 Updated by Matt Travis about 9 years ago

Its the initial hang. I've just tested it with the dataset above and still get a lag between clicking add new feature and being able to digitise. I'm using the nightly download of dev from the 5/10/15.

#15 Updated by Saber Razmjooei about 9 years ago

  • Status changed from Feedback to Open

I think it will be good to give user feedback during the initial indexing. You can see in this screencast, nothing is happening between 0:57 and 1:25 and then the polygon capture becomes active:
https://youtu.be/Hmx_kTKzfRo

#16 Updated by Nyall Dawson about 9 years ago

  • Subject changed from Snapping to layer function: memory use grows indefinitely to Large pause before snapping to large layer

#17 Updated by Saber Razmjooei about 9 years ago

I have done some more test and it seems it has something to do with the data provider as well as size of the file:
  • Using a PostGIS layer (os mastermap) with 1034924 features, the initial lag was too long and QGIS became completely unusable my PC (32 GB RAM, 1 TB SSD, PG layer is from localhost)
  • Saving the same layer as Spatialite, things improved significantly. There was an initial lag, but after that it was smooth digitising.

One can replicate it using OSM as PG layer and Spatialite layer.

#18 Updated by Martin Dobias almost 9 years ago

  • Status changed from Open to Closed

#19 Updated by Matt Travis almost 9 years ago

Just tested with 2.13.0-83 and it works for me. Takes about 6 seconds now so a vast improvement from 60+

Thanks for the fix!

Also available in: Atom PDF