Juan Pablo Tosca Villanueva (Jul 25 2024 at 20:19): Juan Pablo Tosca Villanueva (Jul 25 2024 at 20:19):@Don Sizemore I was looking at this https://github.com/IQSS/dataverse-metrics/issues/92 :eyes: do you have this on your radar? lol
Philip Durbin 🚀 (Jul 25 2024 at 20:41):I'm guilty of making that PR. :sweat_smile:
Don Sizemore (Jul 26 2024 at 12:24):@Juan Pablo Tosca Villanueva the history here is that, maybe a year or two ago(? too lazy to look) the global and site-specific metrics codebases were split apart, and global hasn't seen much attention since then. sounds like Stefano wants some metrics work done in general, but yes it's on my radar.
Juan Pablo Tosca Villanueva (Jul 26 2024 at 12:24):I was wondering if we should merge that pr lol!
Juan Pablo Tosca Villanueva (Jul 26 2024 at 12:25):It has a review pending from you so I was not sure :stuck_out_tongue_wink:
Juan Pablo Tosca Villanueva (Jul 26 2024 at 12:26):Before we pull a crowdstrike :rolling_on_the_floor_laughing::rolling_on_the_floor_laughing::rolling_on_the_floor_laughing:
Philip Durbin 🚀 (Jul 26 2024 at 13:04):I'm in favor of merging it (of course, as the author), but there's no need to deploy it right away. We could even make a release first.
Juan Pablo Tosca Villanueva (Jul 26 2024 at 13:06):I am down to merge it but since it was assigned to @Don Sizemore wanted to ask first lol
Don Sizemore (Jul 26 2024 at 13:07):I did review it? https://github.com/IQSS/dataverse-metrics/pull/101
Don Sizemore (Jul 26 2024 at 13:08):Screenshot-2024-07-26-at-09.07.47.png
Juan Pablo Tosca Villanueva (Jul 26 2024 at 13:08):Oh!
Juan Pablo Tosca Villanueva (Jul 26 2024 at 13:08)::see_no_evil:
Don Sizemore (Jul 26 2024 at 13:09):let me know when you want me to update the code served from metrics.dataverse.org and I will.
Oliver Bertuch (Jul 26 2024 at 13:10):@Don Sizemore Beware, the review comment says "pending", so you did create a draft that is not visible to someone else.
Philip Durbin 🚀 (Jul 26 2024 at 13:10):I'm happy to put our a release first, if you want. Maybe it would be a good occasion to bump to 0.3.
Juan Pablo Tosca Villanueva (Jul 26 2024 at 13:10):I merged it!
Don Sizemore (Jul 26 2024 at 14:15):@Oliver Bertuch why in the absolute hockey sticks would GitHub create a review which is not visible to the submitting party? my only options alongside that comment are to edit/delete/report.
Philip Durbin 🚀 (Jul 26 2024 at 14:47):edit/delete/report? Should I change something?
Philip Durbin 🚀 (Aug 07 2024 at 19:20):@Don Sizemore do you happen to remember seeing this?
<urlopen error [Errno 54] Connection reset by peer>
https://borealisdata.ca had an oops: <urlopen error [Errno 54] Connection reset by peer>
Nothing is jumping out at me at https://github.com/IQSS/dataverse-metrics/issues
Don Sizemore (Aug 07 2024 at 19:41):@Philip Durbin I opened https://github.com/IQSS/dataverse-metrics/issues/97 in response to Borealis disallowing .. but also they're behind a security device that will block anything it doesn't like
Don Sizemore (Aug 07 2024 at 19:41):last time, I wrote to Bikram to get metrics unblocked, I think they blocked all of 3.n.n.n
Philip Durbin 🚀 (Aug 07 2024 at 19:45):@Bikram help! :grinning:
Philip Durbin 🚀 (Aug 07 2024 at 19:47):I pushed a few unrelated commits for stuff I missed in https://github.com/IQSS/dataverse-metrics/pull/101 (restore to global only).
Juan Pablo Tosca Villanueva (Aug 07 2024 at 19:53):We may have found an installation that is missing from the list in production
Juan Pablo Tosca Villanueva (Aug 07 2024 at 19:53): Juan Pablo Tosca Villanueva (Aug 07 2024 at 19:53):Is not responding
Juan Pablo Tosca Villanueva (Aug 07 2024 at 19:54):From @Don Sizemore [rocky@ip-10-0-2-188 2024-08]$ ls -al abertos
ls: cannot access 'abertos': No such file or directory
Bikram (Aug 07 2024 at 19:54):thanks for tagging Phil, how can I help? is it getting blocked again by borealis?
Don Sizemore (Aug 07 2024 at 19:54):@Juan Pablo Tosca Villanueva it's in the config.json list, but it's not responding so there is no cached file.
Don Sizemore (Aug 07 2024 at 19:55):@Bikram Juan Pablo is working with metrics and getting a 403 from Borealis
Bikram (Aug 07 2024 at 19:57):O okay, let me check
Philip Durbin 🚀 (Aug 07 2024 at 20:00):Thanks!
Juan Pablo Tosca Villanueva (Aug 07 2024 at 20:01):Thanks a lot @Bikram
Bikram (Aug 07 2024 at 20:24):I see only requests with 403 from these 2 IPs "96.230.239.124" "65.112.8.22"
none of these IPs is in block list. I am guessing its because of the specific request "GET /api/info/metrics/../version"
Bikram (Aug 07 2024 at 20:25):I think we have a rule in our proxy which does not allow "/../" In requests.
Juan Pablo Tosca Villanueva (Aug 07 2024 at 20:28):I see! I will give this a look, thanks for the information @Bikram t
Bikram (Aug 07 2024 at 20:30):I see IP "65.112.8.22" is gathering metrics from borealis right now.
Philip Durbin 🚀 (Aug 07 2024 at 20:30):Yikes, that's me. Sorry!
Philip Durbin 🚀 (Aug 07 2024 at 20:31):I removed "../version" from this but I still saw the error:
"endpoints": {
"single": [
"dataverses/byCategory",
"datasets/bySubject",
"../version"
],
Actually, huh. Adding time.sleep(1) seems to be helping. (Plus I still have the ../version endpoint out.)
Bikram (Aug 07 2024 at 20:38):oh I see now, its being blocked by abuse rule :grimacing:
Bikram (Aug 07 2024 at 20:39):0x2f12180: key=65.112.8.22 use=0 exp=59324 shard=0 gpc0=0 conn_rate(3000)=2 conn_cur=0 http_req_rate(10000)=8 http_err_rate(20000)=0
sorry for the confusion :pensive:
Bikram (Aug 07 2024 at 20:42):due to recent AI boom and everyone want to crawl data with bots, we have been rate limiting access to one our main service with lots of data https://journals.scholarsportal.info/ and Borealis is behind same proxy.
@Philip Durbin your method is perfect, just add a bit of delay and proxy is happy. And you can add ../version back
Philip Durbin 🚀 (Aug 07 2024 at 20:44):I dunno, I'm still seeing this:
https://borealisdata.ca/api/info/metrics/../version
https://borealisdata.ca had an oops: HTTP Error 403: Forbidden
Smells like a bug. I agree we shouldn't be sending that ...
Bikram (Aug 07 2024 at 20:48):when I do curl or use a browser to access https://borealisdata.ca/api/info/metrics/../version, the request is automatically changing to GET /api/info/version
I think its the bug with python library not transforming the request. And I was right about blocking requests with /../
Philip Durbin 🚀 (Aug 07 2024 at 20:59):Meh, I say it's a bug in our code. Here's a quick fix to get rid of the ..:
https://github.com/IQSS/dataverse-metrics/pull/103
Philip Durbin 🚀 (Aug 07 2024 at 20:59):@Juan Pablo Tosca Villanueva if you want to make a PR to add some sleep, please feel free.
Philip Durbin 🚀 (Aug 07 2024 at 21:00):We seem to need both fixes.
Philip Durbin 🚀 (Aug 07 2024 at 21:00):@Bikram thanks for all your help!
Juan Pablo Tosca Villanueva (Aug 07 2024 at 21:00):Great! Want me to merge it?
Bikram (Aug 07 2024 at 21:02):you are welcome, happy to help :smile:
Philip Durbin 🚀 (Aug 08 2024 at 00:10):Sure! Go for it!
Juan Pablo Tosca Villanueva (Aug 08 2024 at 14:01):keen.zih.tu-dresden.de seems to also be down
Philip Durbin 🚀 (Aug 08 2024 at 14:04):Huh. I guess you could create an issue for that too.
Juan Pablo Tosca Villanueva (Aug 08 2024 at 14:09):Where should I create the issue? both http://dadosabertos.rnp.br/ and http://keen.zih.tu-dresden.de/ are down or should we contact someone in there?
Juan Pablo Tosca Villanueva (Aug 08 2024 at 14:09):on the dv repo?
Philip Durbin 🚀 (Aug 08 2024 at 14:09):I would think https://github.com/IQSS/dataverse-metrics/issues
Philip Durbin 🚀 (Aug 08 2024 at 14:10):Or https://github.com/IQSS/dataverse-installations/issues
Juan Pablo Tosca Villanueva (Aug 08 2024 at 14:10):Thanks :smile:
Philip Durbin 🚀 (Aug 08 2024 at 14:10):For example https://github.com/IQSS/dataverse-installations/issues/235 or https://github.com/IQSS/dataverse-installations/issues/211
Philip Durbin 🚀 (Aug 08 2024 at 14:10):We can always move the issue to another repo.
Don Sizemore (Aug 12 2025 at 11:59):Philip Durbin 🚀 said:
Don Sizemore do you happen to remember seeing this?
<urlopen error [Errno 54] Connection reset by peer> https://borealisdata.ca had an oops: <urlopen error [Errno 54] Connection reset by peer>Nothing is jumping out at me at https://github.com/IQSS/dataverse-metrics/issues
known issue in the older metrics code. the dataverse metrics code makes use of /../ in the URL to construct the info API call, and Borealis' WAF chomps it: https://github.com/IQSS/dataverse-metrics/issues/97 (are y'all using the "global" metrics code?)
Philip Durbin 🚀 (Aug 12 2025 at 13:19):Sorry who are we talking about using the "global" metrics code?
Also, thanks for reminding me of that issue.
Don Sizemore (Aug 12 2025 at 14:22):@Philip Durbin 🚀 metrics.dataverse.org is still running the (now-deleted) global/ subdirectory version of dataverse-metrics. I can cobble in your PR there.
Philip Durbin 🚀 (Aug 12 2025 at 14:23):Or I can put out a proper release if you like!
Last updated: Nov 01 2025 at 14:11 UTC