Stream: troubleshooting

Topic: Enable Make Data Count

Don Richards (Sep 20 2023 at 14:03):

Trying to setup the Make-Data-Count plugin (not sure the terminology here). I've tried installing different python versions (3.7.17 - 3.9.18) and I keep getting the same error

$ sudo su - counter -c "cd /usr/local/counter-processor-0.1.04/ && CONFIG_FILE=counter-processor-config.yaml python3 main.py"

 Traceback (most recent call last):
  File "main.py", line 2, in <module>
    import config
  File "/usr/local/counter-processor-0.1.04/config/__init__.py", line 1, in <module>
    from .config import *
  File "/usr/local/counter-processor-0.1.04/config/config.py", line 3, in <module>
    from models import *
  File "/usr/local/counter-processor-0.1.04/models/__init__.py", line 2, in <module>
    from .db_actions import DbActions
  File "/usr/local/counter-processor-0.1.04/models/db_actions.py", line 4, in <module>
    from .log_item import LogItem
  File "/usr/local/counter-processor-0.1.04/models/log_item.py", line 9, in <module>
    class LogItem(BaseModel):
  File "/home/counter/.local/lib/python3.7/site-packages/peewee.py", line 6172, in __new__
    cls._meta.add_field(name, field)
  File "/home/counter/.local/lib/python3.7/site-packages/peewee.py", line 5961, in add_field
    field.bind(self.model, field_name, set_attribute)
  File "/home/counter/.local/lib/python3.7/site-packages/peewee.py", line 4728, in bind
    self._db_hook(model._meta.database)
  File "/home/counter/.local/lib/python3.7/site-packages/peewee.py", line 4720, in _db_hook
    self._constructor = database.get_binary_type()
  File "/home/counter/.local/lib/python3.7/site-packages/peewee.py", line 3680, in get_binary_type
    return sqlite3.Binary
AttributeError: 'NoneType' object has no attribute 'Binary'

I'm not sure what I'm doing wrong here. It seems like running the main.py command results in this peewee error. I've tried updating peewee as well with the same error. Sorry if this isn't the wrong place to post this. I just signed up for an account today and haven't had time to review what others are posting.

Philip Durbin 🚀 (Sep 20 2023 at 14:05):

What version of Dataverse are you running?

Don Richards (Sep 20 2023 at 14:18):

v. 5.13 build 1244-79d6e57

Philip Durbin 🚀 (Sep 20 2023 at 14:20):

Thanks, I want to see what we say at https://guides.dataverse.org/en/5.13/installation/prerequisites.html#counter-processor

Philip Durbin 🚀 (Sep 20 2023 at 14:20):

Is this on Rocky? Ubuntu? What version, please.

Don Richards (Sep 20 2023 at 14:22):

I can give my exact steps. I wrote them into a script if that helps. The script more or less followed that page. It's RHEL8.8.

Don Richards (Sep 20 2023 at 14:23):

https://gist.github.com/DonRichards/44b5cd090606fa72a1c7ef5b692f9e47

Philip Durbin 🚀 (Sep 20 2023 at 14:24):

If it helps, this is what we do in Ansible: https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible/blob/7d8a425a8e1293717f31c2fd6d01c1d1801cf625/tasks/dataverse-counter.yml

Philip Durbin 🚀 (Sep 20 2023 at 14:25):

It looks like we use the same version, 0.1.04: https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible/blob/7d8a425a8e1293717f31c2fd6d01c1d1801cf625/defaults/main.yml#L141

Don Richards (Sep 20 2023 at 14:37):

I'll take a look

Philip Durbin 🚀 (Sep 20 2023 at 14:38):

We can agree that ensure pip is a little weird, right? I'm sure it was me that added it before, I got into my pattern of python3 -m venv venv. Is that what you do? venv?

Don Richards (Sep 20 2023 at 14:39):

I didn't but I could. I normally only use venv for development (locally) and not in production.

Philip Durbin 🚀 (Sep 20 2023 at 14:39):

I see. What do you do in production? When you encounter a requirements.txt file?

Philip Durbin 🚀 (Sep 20 2023 at 14:42):

Wow, quite a list: https://github.com/CDLUC3/counter-processor/blob/v0.1.04/requirements.txt

Philip Durbin 🚀 (Sep 20 2023 at 14:42):

All versions seem to be pinned, at least. That's good.

Philip Durbin 🚀 (Sep 20 2023 at 14:46):

From your gist, it looks like you're doing this:

sudo -u counter python3 -m pip install -r requirements.txt

Philip Durbin 🚀 (Sep 20 2023 at 14:46):

Plus installing PyYAML and upgrading peewee (like you mentioned).

Philip Durbin 🚀 (Sep 20 2023 at 14:47):

Do you want me to try it your way?

Philip Durbin 🚀 (Sep 20 2023 at 14:48):

I think I would stick with whatever version of peewee (whatever that is) from requirements.txt file.

Philip Durbin 🚀 (Sep 20 2023 at 14:48):

And I'm not sure why you're installing PyYAML. Should it be in requirements.txt?

Philip Durbin 🚀 (Sep 20 2023 at 14:49):

Oh. It is already. PyYAML==5.3.1

Philip Durbin 🚀 (Sep 20 2023 at 14:54):

Basically, I don't want to bother with venv if it isn't a good test.

Don Richards (Sep 20 2023 at 15:30):

It was a result of troubleshooting (manually adding those).

Philip Durbin 🚀 (Sep 20 2023 at 15:32):

That's what I figured. :big_smile:

Don Richards (Sep 20 2023 at 15:32):

I removed them and moved the logic into a venv. Reviewing your Ansible script I realize that I had incorrect permissions for the logs and a typo in a sed command.

Philip Durbin 🚀 (Sep 20 2023 at 15:32):

Oh! Well, all credit to @Don Sizemore and @Péter Pallinger for the Ansible stuff!

Don Richards (Sep 20 2023 at 15:33):

Now I'm getting a "Permission denied: 'state/statefile.json'" but I'm not sure where that's at

Philip Durbin 🚀 (Sep 20 2023 at 15:33):

Sounds like you're a lot closer.

Don Richards (Sep 20 2023 at 15:33):

I've been talking with DonS. We have him as a contractor.
Yes, thanks for the help

Philip Durbin 🚀 (Sep 20 2023 at 15:35):

awesome

Philip Durbin 🚀 (Sep 20 2023 at 15:35):

We do mention statefile.json here: https://guides.dataverse.org/en/5.13/developers/make-data-count.html#testing-make-data-count-and-your-dataverse-installation (dev guide)

Don Richards (Sep 20 2023 at 15:50):

Thanks, that's helpful

Don Richards (Sep 21 2023 at 14:28):

...

Don Richards (Sep 29 2023 at 14:04):

@Philip Durbin Is there a simple way to check to see if make-data-count is working? The log files are not 0 bytes and the api returns without errors but also without data.

Philip Durbin 🚀 (Sep 29 2023 at 14:36):

No data?!?

Philip Durbin 🚀 (Sep 29 2023 at 14:37):

Did you feed the logs files into Counter Processor?

Don Richards (Oct 03 2023 at 15:12):

Yes, that part "seems" to be working.

Philip Durbin 🚀 (Oct 03 2023 at 15:19):

"seems" :grinning:

Don Richards (Oct 03 2023 at 15:22):

Yeah, I have no idea what I'm looking at.

Don Richards (Oct 03 2023 at 15:28):

Normally I'd know exactly what a system trying accomplish and how it works so I can better identify issues. But Dataverse is mostly new to me and I was handed this project in the middle of a massive upgrade to another system that I wasn't familiar with. Now I'm trying to refocus on this. And I'm a bit lost. My best understanding is that MDC is looking at the logs and creating a type of in-house analytics. I pointed the configs to the wrong logs initially and all I got was zero byte logs from MDC. Once I changed the pointer to the correct location of the logs the MDC logs started getting bigger and the API calls were no longer seeming to respond with errors.

Philip Durbin 🚀 (Oct 03 2023 at 16:44):

Does the diagram at https://guides.dataverse.org/en/6.0/admin/make-data-count.html#architecture help?

Philip Durbin 🚀 (Oct 03 2023 at 16:44):

You might need to open the image in a new tab to read it: https://guides.dataverse.org/en/6.0/_images/make-data-count.png

Don Richards (Oct 03 2023 at 19:25):

Double checking a few things. Is this line correct in my config file or am I supposed to change it to something specific?

log_name_pattern: '/usr/local/payara5/glassfish/domains/domain1/logs/counter_(YYYY-MM-DD).log'

Ownership of logs/ is counter :dataverse
The demo says payara6 but we have version 5. Is that a problem?
I also set the output_file to this

output_file: /usr/local/counter-processor-0.1.04/tmp/make-data-count-report

with the tmp/ directory ownership is also counter:dataverse
And to run a test I'm running this

sudo -u counter sh -c "CONFIG_FILE=$COUNTER_PROCESSOR_DIRECTORY/counter-processor-config.yaml YEAR_MONTH=$YEAR_MONTH venv/bin/python3 main.py >> /usr/local/counter-processor-0.1.04/tmp/counter_daily.log"
run counter-processor as counter user
Traceback (most recent call last):
  File "main.py", line 51, in <module>
    main()
  File "main.py", line 29, in main
    with open(lf) as infile:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/payara5/glassfish/domains/domain1/logs/counter_(YYYY-MM-DD).log'

Don Richards (Oct 03 2023 at 19:47):

The image really didn't help clarify for me. I guess I need to know exactly what folders & files to create, what exact permissions need to be set for the different directories and files, what exactly do I need to modify for the config file & what is the expected file path for the config file, am I correct on my location for the make-data-count-report.json file, does the output_file in the config need to be an absolute path or can it be relative, and what exactly is addUsageMetricsFromSushiReport doing?

Philip Durbin 🚀 (Oct 03 2023 at 20:15):

MDC should work fine on Payara 5.

Philip Durbin 🚀 (Oct 03 2023 at 20:19):

What if you do lowercase like this?

log_name_pattern: /usr/local/payara5/glassfish/domains/domain1/logs/mdc/counter_(yyyy-mm-dd).log

Philip Durbin 🚀 (Oct 03 2023 at 20:20):

yyyy-mm-dd instead of YYYY-MM-DD, I mean

Don Richards (Oct 06 2023 at 17:14):

Basically the same. All of the files in that directory are zero bytes. Not sure if that means anything.

Traceback (most recent call last):
  File "main.py", line 51, in <module>
    main()
  File "main.py", line 29, in main
    with open(lf) as infile:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/payara5/glassfish/domains/domain1/logs/mdc/counter_(YYYY-MM-DD).log'

Don Richards (Oct 13 2023 at 20:58):

When I switched to the parent directory it's the same error but the files aren't empty.

Traceback (most recent call last):
  File "main.py", line 51, in <module>
    main()
  File "main.py", line 29, in main
    with open(lf) as infile:
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/payara5/glassfish/domains/domain1/logs/counter_(YYYY-MM-DD).log'

Philip Durbin 🚀 (Oct 13 2023 at 20:59):

Bah, it's quittin' time here but please remind me next week.

Don Richards (Oct 16 2023 at 14:35):

Just a reminder

Philip Durbin 🚀 (Oct 16 2023 at 14:55):

Ha, thanks. Standup in 5 minutes and other meetings but I'll try to circle back.

Don Richards (Oct 16 2023 at 14:56):

No problem. I have meetings today as well, feel free to take your time and don't feel obligated. Thanks for all of the help

Philip Durbin 🚀 (Oct 16 2023 at 18:22):

Is it crazy talk to ask for a shell on your server?

Don Richards (Oct 16 2023 at 19:11):

lol. It would be super helpful to know all of the folder permission (which I did kinda get from your Ansible Scripts) and where everything needs to be. A simple script to check configurations and/or some automated feedback would have been helpful during this process.

Don Richards (Oct 16 2023 at 19:17):

I'm not sure if this helps but I wrote this script to do everything to set this server up. It also shows the counter_dail.sh script I'm using to test this. https://gist.github.com/DonRichards/44b5cd090606fa72a1c7ef5b692f9e47

Philip Durbin 🚀 (Oct 16 2023 at 19:29):

Right. I remember that gist. I'm just thinking it might be nice if we were on the same server. Would you want to spin one up that we could both ssh to? And use dataverse-ansible to install Counter Processor on?

Don Richards (Oct 16 2023 at 20:09):

JHU has so many layers of security it's complicated to get anything outside of the firewall.

Philip Durbin 🚀 (Oct 16 2023 at 20:35):

Yeah, I was thinking about an EC2 instance on AWS.

Oliver Bertuch (Oct 16 2023 at 20:36):

Just in case that makes things easier to expose to outside: https://butlerx.github.io/wetty/

Don Richards (Oct 16 2023 at 21:04):

I'll talk to my system admin in the morning and see if it's possible with our setup

Philip Durbin 🚀 (Oct 16 2023 at 21:06):

Thanks. And who else in the community has set MDC up, I wonder. It's been a while for me. :sweat_smile:

Philip Durbin 🚀 (Oct 17 2023 at 18:00):

I'm trying to spin up an EC2 instance with MDC installed by dataverse-ansible.

Philip Durbin 🚀 (Oct 18 2023 at 15:52):

Hmm, it's half-configured: http://dev2.dataverse.org

Philip Durbin 🚀 (Jan 24 2024 at 15:21):

@Don Richards it looks like @luddaniel is setting up MDC, Counter Processor, etc. Please see #community > Make Data Count

Maybe he can help!

Did you ever get it working? :sweat_smile:

Don Richards (Jan 24 2024 at 15:54):

That's interesting. Thanks.

Philip Durbin 🚀 (Jan 24 2024 at 15:57):

Sure. Sorry this has been so difficult.

Last updated: Oct 30 2025 at 06:21 UTC