Bug report #12933

Heatmap generating an empty raster

Added by David Vanescentes over 9 years ago. Updated almost 6 years ago.

Status:Closed
Priority:Low
Assignee:-
Category:Analysis library
Affected QGIS version:master Regression?:No
Operating System: Easy fix?:No
Pull Request or Patch supplied:No Resolution:end of life
Crashes QGIS or corrupts data:No Copied to github as #:21015

Description

I am trying to create a heatmap raster using a point layer loaded from sql server. This is working with some settings but many settings result in an empty raster (at least when viewed in QGIS Browser (I guess this could be an issue not with the generation but the rendering of the generated raster layer because the file it outputs is not empty). I load the point layer ("point_data" table in the attached database backup), select "Raster" menu item, then "Heatmap", then "Heatmap...". I then set the settings as the appear in the attached screenshot called "heatmap-dialog.gif". The result (after a the progress bar is displayed for a little bit) can be seen in the screenshot called "result.gif" (raster appears to not have any data in it or to not be rendered). I don't see any entries in the log window tabs about this ("View"..."Panels"..."Log Messages").

heatmap-dialog.gif - heatmap dialog screenshot (16.5 KB) David Vanescentes, 2015-06-10 03:06 PM

result.gif - result after attempting to create heatmap raster (141 KB) David Vanescentes, 2015-06-10 03:06 PM

heatmap_data.bak.001 - database backup (part 1 of 2) (5 MB) David Vanescentes, 2015-06-10 03:10 PM

heatmap_data.bak.002 - database backup (part 2 of 2) (1.13 MB) David Vanescentes, 2015-06-10 03:12 PM

heatmap_raster24.tif (2.31 MB) David Vanescentes, 2015-06-10 03:29 PM

shapefile.zip - ESRI Shapefile version of the dataset (same dataset represented in the database backup files already uploaded) (2.86 MB) David Vanescentes, 2015-06-11 01:30 PM

points.png (162 KB) Giovanni Manghi, 2015-06-24 08:25 AM

sample-points-with-country-shapes.gif (162 KB) David Vanescentes, 2015-06-24 09:07 AM

shapefile_limited.zip (2.72 MB) David Vanescentes, 2015-06-24 10:07 PM

History

#1 Updated by David Vanescentes over 9 years ago

It didn't include my database backup file because it was too big I guess (6 MB). I am splitting the file up into chunks no bigger than 5MB with HJ Split (you can join this file and the next one with hj split: http://www.hjsplit.org/).

#2 Updated by David Vanescentes over 9 years ago

This is part 2 of the database backup file. Join this with "heatmap_data.bak.001" using HJ Split (or some other program that will concatenate files).

#3 Updated by David Vanescentes over 9 years ago

I should upload the geotiff raster file it produces as well.

In addition, it might be useful to know that it does work for me when I use a radius of 0.5 map units (instead of 0.1). It also does work with a smaller dataset using a 0.1 radius (a subset of this dataset of 100 points). So I guess that sounds like it might be a memory issue. My PC has 16 GB of memory. When using meters when defining the radius it also cannot handle smaller radii (like 20000 meters does not work but 60000 does work).

#4 Updated by Giovanni Manghi over 9 years ago

  • Category changed from Browser to 114
  • Status changed from Open to Feedback
  • Target version deleted (Version 2.10)

does the outcome changes if you use a different data format? if the result is the same can you attach (or link, or share) the dataset?
Can you please ALSO test qgis master? Thanks.

#5 Updated by David Vanescentes over 9 years ago

Thanks for looking at this.

In QGIS, I saved the point layer that I loaded from Sql Server as an ESRI shapefile. I then closed QGIS completely and loaded that shapefile and tried the steps described in my original post and QGIS behaved the same way it did when I loaded the data from sql server. I am attaching the shapefile files zipped in a file called "shapefile.zip".

#6 Updated by Giovanni Manghi over 9 years ago

  • Affected QGIS version changed from 2.8.2 to master
  • Priority changed from Normal to Low
  • Operating System deleted (Windows)
  • OS version deleted (Windows 8.1 64 bit)

In addition, it might be useful to know that it does work for me when I use a radius of 0.5 map units (instead of 0.1). It also does work with a smaller dataset using a 0.1 radius (a subset of this dataset of 100 points). So I guess that sounds like it might be a memory issue. My PC has 16 GB of memory. When using meters when defining the radius it also cannot handle smaller radii (like 20000 meters does not work but 60000 does work).

I don't think it is a memory issue. In fact with smaller search radius the operation is easier. Here the culprit seems to be the very odd position of a few of the points of the dataset. I say odd because they don't seem right at all (for too north to be right). If I remove such points then the heatmap is correctly created. I would say that this is ticket has to be closed as it is about a very particular dataset, that probably is partially wrong.

#7 Updated by David Vanescentes over 9 years ago

I am not sure what you mean when you said some points in the dataset were too far north. I randomized the dataset without care to where the points were landing so some were pretty far north...but I did ensure that they were between -90 and 90 degrees latitude and -180 and 180 degrees longitude. Which points did you delete to make it work? Is it possible you deleted a large portion of the points and the bug was not triggered because of the reduced dataset size? I have noticed that on smaller datasets I do not have this issue (at least the few smaller datasets I have used).

#8 Updated by David Vanescentes over 9 years ago

I really don't think this issue is due to quirks in this specific dataset (other than its size maybe) so I think it is a mistake to drop the priority of this issue or to entirely discount this bug. If it'd help you, I am happy to make a random set of data points that fall within a tighter bounding box (not so far north) but still trigger the bug ... even though that seems like it would be a wasted effort unless for some reason the dataset I originally posted has something actually invalid about it (points are not invalid because they happen to be in the northern hemisphere right?). Let me know what you need to proceed.

Thanks,

David

#9 Updated by Giovanni Manghi over 9 years ago

David Vanescentes wrote:

I really don't think this issue is due to quirks in this specific dataset (other than its size maybe) so I think it is a mistake to drop the priority of this issue or to entirely discount this bug. If it'd help you, I am happy to make a random set of data points that fall within a tighter bounding box (not so far north) but still trigger the bug ... even though that seems like it would be a wasted effort unless for some reason the dataset I originally posted has something actually invalid about it (points are not invalid because they happen to be in the northern hemisphere right?). Let me know what you need to proceed.

Thanks,

David

I don't know, see the attached image. Removing those (maybe) wrong points make things work here...

#10 Updated by David Vanescentes over 9 years ago

I think you might not be aligning that bing picture correctly or maybe something else is going awry with how you are loading "shapefile.shp" (I assume you are using shapefile.shp found in the attached file called "shapefile.zip"). The max y coordinate (latitude) in that shapefile is 89.3696. In QGIS desktop, I load a country shapes layer and then load the shapefile (shapefile.shp) and I see what is in the attached image (which appears to correctly show the data and is a little different distribution of points than the image you posted..."points.png").

#11 Updated by David Vanescentes over 9 years ago

After I posted that last post I thought about it and I have a theory as to why your image (points.png) shows the points very far from where they should be on the bing maps image. I am a total newbie with GIS so this could be wrong but since, in your picture, the points seem to be stretched longitudinally along certain longitudes, I think you may be projecting the points using a different projection than the one used to create the bing maps image you included (like maybe the bing maps image was created using Web Mercator and you projected the points with Goode homolosine or something)

#12 Updated by David Vanescentes over 9 years ago

I decided to create a version of the shapefile where there are no points over 72.18 degrees latitude (moved those points above into the southern hemisphere so there are the same number of points but it doesn't extend too far north. With this shapefile (called shapefile_limited.shp) the same behavior occurs for me that I described for the original dataset.

#13 Updated by Giovanni Manghi over 9 years ago

David Vanescentes wrote:

I decided to create a version of the shapefile where there are no points over 72.18 degrees latitude (moved those points above into the southern hemisphere so there are the same number of points but it doesn't extend too far north. With this shapefile (called shapefile_limited.shp) the same behavior occurs for me that I described for the original dataset.

your shape is in wgs84, so map units are degrees, are you using a proper search radius, like (example) 2/3?

#14 Updated by David Vanescentes over 9 years ago

What do you mean when you say "are you using a proper search radius"? Like I wrote above, I tried using a radius of 0.1 map units (which are degrees in this case) and the raster comes up blank. When I use 0.5 map units (degrees), it successfully creates the raster. The same thing happens when you use meters to define smaller or larger radii (for example it fails with 20,000 meters and succeeds when using 60,000 meters).

Have you tried recreating the issue with the files or data I attached? Let me know if you have any trouble seeing the behavior/bug I described.

#15 Updated by David Vanescentes over 9 years ago

Can you move the priority back up now that it is clear your reasoning for dropping it was wrong (that the issue was due to some odd quirk specific to the dataset I was using)? If you are not convinced that this issue manifests itself for a wide range of dataset/shapefiles, tell me how many randomized shapefiles you want to see that result in the same issue and I will generate that many.

#16 Updated by David Vanescentes over 9 years ago

  • Assignee set to Nathan Woodrow

Due to the lack of response I think this may have been abandoned (frustrating).

#17 Updated by Nathan Woodrow over 9 years ago

  • Assignee deleted (Nathan Woodrow)

I don't think it has been abandoned, it's only 2 months old and we have a lot of other things on. I'm not the right person to fix this as it's outside of my knowledge area.

I understand it's frustrating at times however a lot of people put a lot of free time into this project and can't jump onto every bug that comes up.

#18 Updated by Giovanni Manghi about 9 years ago

  • Status changed from Feedback to Open

Nathan Woodrow wrote:

I don't think it has been abandoned, it's only 2 months old and we have a lot of other things on. I'm not the right person to fix this as it's outside of my knowledge area.

I understand it's frustrating at times however a lot of people put a lot of free time into this project and can't jump onto every bug that comes up.

it is also needed to say that personally I'm convinced that there are no issues here. 0,1 degrees as search radius for the provided dataset is a too small value and then the output is obviously empty.

#19 Updated by David Vanescentes about 9 years ago

I don't understand what you are saying here Giovanni.

You say "0,1 degrees as search radius for the provided dataset is a too small value and then the output is obviously empty". Are you saying that if the search radius is too small for datasets with points spread over a large area, then there is a bug in QGIS that causes it to fail when producing a raster (and so it just outputs a blank tiff)? If so, that sounds like an actual issue to me.

Are you saying that this behavior is intended (that when a user generates a raster from points, they want a blank raster output if they use "high" resolution settings)? I doubt a user would assume (or want) this behavior.

Are you saying that no 2 points in the dataset are within 0.1 degrees of each other? I am pretty sure this isn't the case but I haven't verified that. I just know that on other datasets I have encountered this problem on, that is definitely not the case.

Are you saying that if no 2 points are close enough to be within each other's search radius, the intended behavior of the raster generator is to output a blank raster? I doubt a user would assume or want this behavior. If you have 1000 points spread apart and you generated a raster from that (with a small search radius), you would want a raster with 1000 little "stars" representing those points.

#20 Updated by Giovanni Manghi almost 8 years ago

  • Category changed from 114 to Analysis library

#21 Updated by Giovanni Manghi almost 8 years ago

  • Target version set to Version 3.0

#22 Updated by Giovanni Manghi over 7 years ago

  • Regression? set to No
  • Easy fix? set to No

#23 Updated by Giovanni Manghi almost 6 years ago

  • Status changed from Open to Closed
  • Resolution set to end of life

Also available in: Atom PDF