QGIS » QGIS Application

Same limit here, 84 layers. Using master build 6d788eb from June 4. And I'm creating a new project, not opening an old one.

From Royce Cline: the limit of the number of open files per process on OS X is 256. I now remember this problem a while back, but figured it was solved somehow. But then, I don't add a lot of shapefiles to a QGIS project, until recently, usually it's postgis connections.

Royce's solution has been to set ulimit in a Terminal and start QGIS from there, but this is not something for the average user, and is generally a bad way (at least the wrong way) to start any application on OS X.

ulimit only affects processes started from a shel (Terminal). OS X apps get their limits from sysctl/launchctl. You can check it with:

launchctl limit

which for me on 10.6 shows 256 soft limit and unlimited hard limit. Huh, if a soft limit only gives the app a warning but lets it continue opening files, maybe there is a problem in how Qt handles it.

The sysctl limits look different:

kern.maxfiles: 12288
kern.maxfilesperproc: 10240

Same issue here. In Linux the limit seems to be 1024 by default. I wonder how other software address this problem since it is platform-specific.

gcarrillo - wrote:

I wonder how other software address this problem since it is platform-specific.

As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.

Giovanni Manghi wrote:

gcarrillo - wrote:

I wonder how other software address this problem since it is platform-specific.

As far as I have understand it is an operating system limitation/issue, not a QGIS one. Consider also that when adding a shapefile you are opening at least 3 files, not 1.

Giovanni, if you read what I wrote, you'll see that I'm aware it's not a QGIS problem but a platform (OS) one. My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.

My question goes in the direction of finding the ways other GIS software tackle this, so that QGIS can also implement their solutions, if there are any.

I guess fist we have to know if other GIS packages overcome this limitation. They do? any open source?

I think that a good solution would be to keep the user informed about OS limitations.

In the GRASS Wiki they report OS limitations to users (see Number of open files limitation): http://grasswiki.osgeo.org/wiki/Large_raster_data_processing

So let's keep the information accessible by users and create/update a wiki page about Know Issues.

Thanks in advance.

Closing related Bug #6404.

I created a wikipage

Known Issues

I think there is no more reason to keep this issue open.

Please, if necessary reopen it.

The original issue (as reported for Mac) has been fixed in master branch with commit d8e5c2f8

It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).

When I added a QgsLogger line to the setrlimit() if block, above the QgsDebugMsg line, to check if the rlimit was changed, it then worked in a release build. I wonder then, since QgsDebugMsg() is a conditional define that does nothing in a release build, then the whole if block gets optimized to nothing since the if block is essentially empty. When I added QgsLogger(), that's a real function, so the if block is then not empty, and it then works. Maybe the setrlimit() part of the if test needs to be put in a variable and the variable tested, so it doesn't get lost in optimization?

A side note: in searching for an example of how to get non-debug messages into the console log so I can check the rlimits in a release build, I noticed that non-debug messages are not getting to console any more. The usual QgsMessageLog::logMessage() that I see in gqisapp.cpp, that showed up in console logs in the past do not now. And I couldn't find any QGIS log file, it doesn't look like there is a default, but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. I had to use QgsLogger::warning() to get anything to show in the console, and that's only when running from a Terminal, directly not with the open command.

It wasn't debug vs. release build that was the problem. It's starting from Terminal vs. starting as a normal application. I started the debug build from a terminal to see the debug output for the rlimit settings. Then I started my hacked release build to also see the console output. But when starting either debug or hacked release build as a normal app, the rlimits are not changed and I can't load more than 84 shapefiles.

So, this bug is not fixed yet.

And same with my old v2.2 - opens more than 84 shapefiles when started from Terminal.

William Kyngesburye wrote:

... I noticed that non-debug messages are not getting to console any more. ... but is only set from the environment in QGIS_LOG_FILE, which can't happen in a normal OS X app startup. ...

I think that has been the case since Lion, but definitely for Mt. Lion and Mavericks. Maybe something to do with sandboxing changes.

I use the following in the .app's Info.plist for my local dev builds:

<key>LSEnvironment</key>
<dict>
  ...
  <key>QGIS_LOG_FILE</key>
  <string>/Users/<my_user>/Library/Logs/QGIS/qgis-dev.log</string>
  ...
</dict>

But this may not be reasonable for the stable version, because there is no shell-type expansion for any of those LSEnvironment values. I suppose you could set that to a log file location writable by all users, but would be best in ~/Library/Logs/QGIS/.

Retesting the file limit issue now.

William Kyngesburye wrote:

It partially works now. It works when I build a debug QGIS, but does not work when building a release (MinSizeRel) QGIS. Tested with new v2.4, also release build for 2.2 (didn't try debug for 2.2).

I can not reproduce the limit issue here. Just downloaded/installed your QGIS 2.4, launched it by double-click and ran the following in PyQGIS console:

for i in range(0,200):
  vlayer = QgsVectorLayer("/path/to/shapefile.shp", "layer {0}".format(i), "ogr")
  QgsMapLayerRegistry.instance().addMapLayer(vlayer)

For which, Activity Monitor shows 600 files loaded. I also saved and closed QGIS. Upon relaunching QGIS, the project was successfully auto-loaded (I have the option set).

I have also tested all my various dev builds here, via double-click. The qgis.log I have set shows the following:

Mac RLIMIT_NOFILE Soft/Hard NEW: 10240 / 10240
Mac RLIMIT_NOFILE Soft/Hard ORIG: 256 / 9223372036854775807

Will have to wait until next week to get access to the 10.7 nightly build VM to test it.

Is there maybe something funky with your Mac's setup? Have you tested on a different Mac as well?

William, I tested with 150 separate shapefiles, via your double-clicked 2.4 release install, to make sure repetitively loading the same shapefile wasn't masking the issue. Same results, everything loads fine, with Activity Monitor reporting 450 open shapefile components.

Grr, something off with my Mac. Compiled on this Mac, OS X 10.8, with 10.7 SDK. Won't load > 84 shapefiles. Same problem in another login. Another Mac, OS X 10.7, it works. Yet another Mac 10.8, works.

I upgraded the system on the problem Mac recently. I can now open more than 84 shapefiles. I never figured what in the old system (and just that Mac) was overriding the limits.