Bug report #21694
Hang/Crash when attempting to drag&drop multiple layers into gpkg
| Status: | Open | ||
|---|---|---|---|
| Priority: | High | ||
| Assignee: | - | ||
| Category: | Browser | ||
| Affected QGIS version: | 3.6.1 | Regression?: | No |
| Operating System: | 1809 17763.402 [x64 Pro] | Easy fix?: | No |
| Pull Request or Patch supplied: | No | Resolution: | |
| Crashes QGIS or corrupts data: | Yes | Copied to github as #: | 29510 |
Description
The time it takes to import a number of layers into a geopackage seems to increase exponentially. For instance, there is no noticeable delay from time of drop to beginning of import at less than 10 layers. Somewhere between 10-20 layers there is a pause of 10s or more. Any more than 20 layers the pause becomes longer. Trying to drag/drop 105 layers results in an unresponsive QGIS GUI with 1 CPU thread pegged, no disk activity, and static (not growing) RAM usage for the QGIS process.
There should be the means to batch import data into a geopackage, especially if one is trying to migrate away from an ESRI environment where large file geodatabases are commonplace. Having to batch things a handful of layers at a time is tedious at best.
Related issues
History
#1
Updated by Giovanni Manghi over 5 years ago
- Priority changed from Normal to High
There should be the means to batch import data into a geopackage, especially if one is trying to migrate away from an ESRI environment where large file geodatabases are commonplace. Having to batch things a handful of layers at a time is tedious at best.
there will be soon an option, I hope (let's see if I'm able to implement it): https://issues.qgis.org/issues/21679
#2
Updated by Brett Carlock over 5 years ago
Giovanni Manghi wrote:
There should be the means to batch import data into a geopackage, especially if one is trying to migrate away from an ESRI environment where large file geodatabases are commonplace. Having to batch things a handful of layers at a time is tedious at best.
there will be soon an option, I hope (let's see if I'm able to implement it): https://issues.qgis.org/issues/21679
That sounds amazing!
I'm not sure the behavior above is directly related though. It should be able to handle the import one by one as it does for # of layers less than about 20, but it is just stuck.
Currently 13min of CPU time at 100% for that core (8% overall CPU usage) and no change in RAM or disk activity. QGIS instance still Not responding...
#3
Updated by Giovanni Manghi over 5 years ago
That sounds amazing!
I'm not sure the behavior above is directly related though. It should be able to handle the import one by one as it does for # of layers less than about 20, but it is just stuck.
Currently 13min of CPU time at 100% for that core (8% overall CPU usage) and no change in RAM or disk activity. QGIS instance still Not responding...
it will have the same performances as the GDAL/OGR command line tool, so you can already test it.
#4
Updated by Nyall Dawson over 5 years ago
- Status changed from Open to Feedback
There's already the "package layers" algorithm in the processing toolbox, which allows for many layers to be added to a geopackage. Does that work OK for you?
#5
Updated by Jürgen Fischer over 5 years ago
- Related to Feature request #21695: Improve behavior of Geopackage Data Import added
#6
Updated by Brett Carlock over 5 years ago
Giovanni Manghi wrote:
That sounds amazing!
I'm not sure the behavior above is directly related though. It should be able to handle the import one by one as it does for # of layers less than about 20, but it is just stuck.
Currently 13min of CPU time at 100% for that core (8% overall CPU usage) and no change in RAM or disk activity. QGIS instance still Not responding...
it will have the same performances as the GDAL/OGR command line tool, so you can already test it.
4hr25m of CPU time on this import and it still is hung. To clarify, this is to import 105 different layers totaling 289mb as shapefiles. There is definitely a bug/limitation in the import to geopackage function. There is no way this should take even a fraction of this speed.
i7-7800x
1TB SSD
16GB DDR4
#7
Updated by Giovanni Manghi over 5 years ago
4hr25m of CPU time on this import and it still is hung
this is with the "package layers" tool?
#8
Updated by Brett Carlock over 5 years ago
Giovanni Manghi wrote:
4hr25m of CPU time on this import and it still is hung
this is with the "package layers" tool?
Drag/drop from the Browser panel. Same method I opened the ticket about.
#9
Updated by Brett Carlock over 5 years ago
Nyall Dawson wrote:
There's already the "package layers" algorithm in the processing toolbox, which allows for many layers to be added to a geopackage. Does that work OK for you?
Thanks for pointing that out. It isn't immediately obvious (I had never come across it, for instance) and can't be called from the Browser panel, but yes, 50s of CPU time to import all 105 layers into the geopackage. This works as intended, though please note that drag/drop from the Browser panel does not.
#10
Updated by Giovanni Manghi over 5 years ago
- Status changed from Feedback to Open