argis_rest file formats?

[ashjolly]

Hi team,

In using the bcdata package for downloading spatial data from the Data Catalogue, I've run into multiple instances where data is present as an 'arcgis_rest' file format. My impression is that this happens when the Data Catalogue is scraping data from Map Hub resources.

For example, the Snow Basin Indices item within the Data Catalogue represents snow data present for polygons within the RFC's Snow Map (I believe...) :

RFC Snow Map Link
https://governmentofbc.maps.arcgis.com/apps/webappviewer/index.html?id=b57800e08e46468bab506f9b9f0cbad6

The resultant Data Catalogue entry is in an "arcgis_rest" format:
https://catalogue.data.gov.bc.ca/dataset/snow-basin-indices

library(bcdata)
bcdata::bcdc_get_record("712d39f3-de6f-4ddf-a5e5-2066be5e4482")
bcdata::bcdc_query_geodata("712d39f3-de6f-4ddf-a5e5-2066be5e4482") %>% dplyr::collect()
bcdata::bcdc_get_data("712d39f3-de6f-4ddf-a5e5-2066be5e4482") %>% dplyr::collect()

I notice that the 'bcdata_available' column is FALSE for all of the resources within this link.

We are going to update the polygons within the Snow Map, which will hopefully be represented within the BC Data Catalogue entry. I'm hoping to be able to point my R script to this entry in the BC Data Catalogue rather than rely on a local copy of the spatial data. Additionally, this is also the same situation for the Drought Polygons.

Thanks for your two cents about any suggestions on how to deal with this situation, and thanks for developing such a useful package!

-Ashlee

[stephhazlitt]

Thanks for this Issue @ashjolly.

Currently {bcdata} searches and returns all the metadata records available in the B.C. Data Catalogue. It can only pull data from a catalogue record where the data resource is stored in the B.C. Data Catalogue itself (bcdc_get_data()) or the data resource is stored in the B.C. Geographic Warehouse (bcdc_query_geodata()), with a few exceptions around file types (e.g. {bcdata} does not work when a data resource is a set of multiple files in a zipped folder).

As you point out above, some of the catalogue metadata records are for Web Apps/User Tools, such as the Snow Map---where the record provides a licence and metadata for the App itself. Rather than scraping data from a Web App, I think a more direct path would be to add the data layer itself (in a non-proprietary or open or common format 😉) as a resource within the record.

[ashjolly]

Thanks Stephanie! Excellent points and clarification - I really appreciate it (and apologies for the delay - I was on leave). I know that GeoBC makes the Web App map from spatial data, so I could see a pathway forward where this initial layer is added to the catalogue, along with the Web App meta data you describe. I'll check in with this regarding the RFC-related resources. Thanks again!

[bevingtona]

It might be worth revisiting this idea of bcdata having functionality for ArcGIS REST formats...

For example:

# remotes::install_github("yonghah/esri2sf")
snow_basins_url <- "https://services6.arcgis.com/ubm4tcTYICKBpist/ArcGIS/rest/services/Snow_Basins_Indices_View/FeatureServer/0"
esri2sf::esri2sf(snow_basins_url) |> mapview::mapview(zcol = "Snow_Basin_Index")

There are 59 publicly published datasets in the BC Data Catalogue that are stored as arcgis_rest.

[boshek]

My take is that while this is in scope for bcdata, the real limitation is that esri2sf is not on CRAN and therefore would at best require a dodgy workaround CRAN policies and at worst impose a fairly sizable maintenance cost.

[ateucher]

Yeah, I came here to say the same thing. Would love to do it but without esri2sf on CRAN it's not really doable... One possible way would be to return the url when it's an esri endpoint, and add some documentation about how to use esri2sf?

[boshek]

I mean @bevingtona could also write a custom parser too. I assume it is "just" some json that the esri2sf is handling. 😜

[stephhazlitt]

+1 for breadcrumbs leading users to esri2sf. It might be worthwhile to see if the authors of esri2sf have a path to CRAN?

I'll reiterate my philosophical objection to supporting spatial formats in the BC Data Catalogue—the open data portal—that are not in an open format. I still think this is akin to the getting a horse off a balcony situation, where the horse should not be there in the first place 🐴.

[bevingtona]

For sure, great comments @stephhazlitt @ateucher @boshek .. I think the reality is that so many are hosting data in this format.

I'll look into an in-house parser ..

[bevingtona]

This one is on CRAN ... arcpullr

# install.packages("arcpullr")
library(arcpullr)
snow_basins_url <-"https://services6.arcgis.com/ubm4tcTYICKBpist/ArcGIS/rest/services/Snow_Basins_Indices_View/FeatureServer/0"
arcpullr::get_spatial_layer(snow_basins_url) |> mapview::mapview(zcol = "Snow_Basin_Index")

[ateucher]

That does potentially change things, let's reopen this

[stephhazlitt]

Agreed @ateucher & @bevingtona. If there is a CRAN package we can import to parse these spatial files and there is bandwidth to author a PR, I am +1 for adding this enhancement.

[ateucher]

@bevingtona if you have time and inclination to do a PR, that would probably expedite this. I can probably get to it some time, but I can't say when.