Bug report #15292
Slow deleting attributes table columns of large shapefiles
| Status: | Closed | ||
|---|---|---|---|
| Priority: | Normal | ||
| Assignee: | - | ||
| Category: | Attribute table | ||
| Affected QGIS version: | 2.18.2 | Regression?: | No |
| Operating System: | Easy fix?: | No | |
| Pull Request or Patch supplied: | No | Resolution: | end of life |
| Crashes QGIS or corrupts data: | No | Copied to github as #: | 23226 |
Description
See #15292-3
I had a shapefile with 481k features and a half gigabyte attribute table. I did not require the majority of the data included, and so in order to make the file more manageable (less storage, faster to work with), I used the editor to delete the unwanted columns. To its credit QGIS did complete the task, unlike LibreOffice which simply crashed, however it took 36 hours* to do so! During the process I realized that part of the performance problem stems from the approach QGIS used to delete the specified attributes i.e; removing them column-by-column, then rewriting and reloading the database each time. For shapefiles where there are a large number of records or attributes, or the majority of such are being shed, it seems it would be much more efficient to create a temporary shapefile and copy over the data the user wishes to preserve, and then replace the original file with the new version.
Similarly, if not used, memory mapping large files could increase performance.
*2.5GHz Core i5-5300 with 8GB RAM on Windows 7
History
#1
Updated by Giovanni Manghi over 8 years ago
- Category changed from Browser to Vectors
- Crashes QGIS or corrupts data set to No
- Status changed from Open to Feedback
- Affected QGIS version set to 2.16.0
- Tracker changed from Feature request to Bug report
Could you point me to a dataset that I could youse to replicate the issue? what version of QGIS are you using?
What do you mean with "editor to delete unwanted columns"?
#2
Updated by belg4mit - over 8 years ago
What do you mean with "editor to delete unwanted columns"?
You missed an important verb there, that phrase isn't the name of something. "I used the [attribute table] editor to delete the unwanted columns."
#3
Updated by Giovanni Manghi almost 8 years ago
- Category changed from Vectors to Attribute table
- Affected QGIS version changed from 2.16.0 to 2.18.2
- Status changed from Feedback to Open
- Subject changed from Copy on write for large shapefiles to Slow deleting attributes table columns of large shapefiles
You missed an important verb there, that phrase isn't the name of something. "I used the [attribute table] editor to delete the unwanted columns."
There are at least two places where you can control the attribute table structure: in vector properties and in the toolbar of the attribute table itself, that is what I wanted to know.
In fact the problem you describe arises in just a specific case when controlling the structure from within the attribute table. Tried with a 1 million features, 1.2GB attribute table shapefile of myself:
- deleting columns and saving from the vector properties ("fields" tab) is fast
- deleting columns from within the attribute table is fast only if the table is showing just a (relatively) small number of features, for instnce because in general options the "show features visible on map" configuration was set
- deleting columns from within the attribute table is dead slow if the table of attributes is showing all records of a very large table, exactly as you described.
#4
Updated by Giovanni Manghi over 7 years ago
- Regression? set to No
- Easy fix? set to No
#5
Updated by Jürgen Fischer over 7 years ago
- Description updated (diff)
#6
Updated by Giovanni Manghi over 5 years ago
- Resolution set to end of life
- Status changed from Open to Closed
End of life notice: QGIS 2.18 LTR
Source:
http://blog.qgis.org/2019/03/09/end-of-life-notice-qgis-2-18-ltr/
QGIS 3.4 has recently become our new Long Term Release (LTR) version. This is a major step in our history – a long term release version based on the massive updates, library upgrades and improvements that we carried out in the course of the 2.x to 3x upgrade cycle.
We strongly encourage all users who are currently using QGIS 2.18 LTR as their preferred QGIS release to migrate to QGIS 3.4. This new LTR version will receive regular bugfixes for at least one year. It also includes hundreds of new functions, usability improvements, bugfixes, and other goodies. See the relevant changelogs for a good sampling of all the new features that have gone into version 3.4
Most plugins have been either migrated or incorporated into the core QGIS code base.
We strongly discourage the continued use of QGIS 2.18 LTR as it is now officially unsupported, which means we’ll not provide any bug fix releases for it.
You should also note that we intend to close all bug tickets referring to the now obsolete LTR version. Original reporters will receive a notification of the ticket closure and are encouraged to check whether the issue persists in the new LTR, in which case they should reopen the ticket.
If you would like to better understand the QGIS release roadmap, check out our roadmap page! It outlines the schedule for upcoming releases and will help you plan your deployment of QGIS into an operational environment.
The development of QGIS 3.4 LTR has been made possible by the work of hundreds of volunteers, by the investments of companies, professionals, and administrations, and by continuous donations and financial support from many of you. We sincerely thank you all and encourage you to collaborate and support the project even more, for the long term improvement and sustainability of the QGIS project.