✔ Need Help with Solr Re-indexing!!!!! · troubleshooting

Stream: troubleshooting

Topic: ✔ Need Help with Solr Re-indexing!!!!!

Sherry Lake (Jun 02 2025 at 18:44):

@Philip Durbin Anyone @Don Sizemore

upgraded to V6.6, but after re-index (index in place), newly created dataset was not showing up in UI. Could get to it via direct link.

So, did index "clear"

curl http://localhost:8080/api/admin/index/clear

and am now doing....

curl http://localhost:8080/api/admin/index

Nothing is showing up in UI:
http://dataverse.lib.virginia.edu

But of course all data is there:

https://dataverse.lib.virginia.edu/dataset.xhtml?persistentId=doi:10.18130/V3/ZNR96G

Sherry Lake (Jun 02 2025 at 18:58):

Indexing individual file is working. And I finally see two unpublished datasets (as dataverseAdmin). But that's all.

Don Sizemore (Jun 02 2025 at 19:13):

Hi Sherry, what shows up in the Payara server.log during re-index? (Have you re-traced your steps through the Solr installation?)

Philip Durbin 🚀 (Jun 02 2025 at 19:14):

Hmm, I see four datasets at https://dataverse.lib.virginia.edu/dataverse/uva and they're all published.

Philip Durbin 🚀 (Jun 02 2025 at 19:16):

I agree with Don that seeing server.log will probably help.

Sherry Lake (Jun 02 2025 at 19:33):

We have a local metadata block....

AND our solr is on another box..... that we MUST put through github and terraform???? or teamcity???
I didn't do that part....

I downloaded both solrconfig.xml and schema.xml.

I ran update-fields.sh (command from the installation instructions) on the downloaded schema.xml.

Then I had to put both solrconfig.xml and schema.xml in version control and over to github... where my sysadmin (Tim), picked up.

He did all that part in addition to the updated solr version.

Sherry Lake (Jun 02 2025 at 19:37):

Here are the most recent SEVERE errors:

[2025-06-02T19:31:43.545+0000] [Payara 6.2025.2] [SEVERE] [] [jakarta.enterprise.resource.webcontainer.faces.context] [tid: _ThreadID=93 _ThreadName=http-thread-pool::jk-connector(4)] [timeMillis: 1748892703545] [levelValue: 1000] [[

jakarta.faces.view.facelets.TagAttributeException: /dataset.xhtml @75,75 test="#{empty DatasetPage.croissant}" /dataset.xhtml @75,75 test="#{empty DatasetPage.croissant}": java.lang.NullPointerException: Cannot invoke "edu.harvard.iq.dataverse.DatasetVersion.isPublished()" because "this.workingVersion" is null

jakarta.faces.view.facelets.TagAttributeException: /dataset.xhtml @75,75 test="#{empty DatasetPage.croissant}" /dataset.xhtml @75,75 test="#{empty DatasetPage.croissant}": java.lang.NullPointerException: Cannot invoke "edu.harvard.iq.dataverse.DatasetVersion.isPublished()" because "this.workingVersion" is null

at com.sun.faces.facelets.tag.TagAttributeImpl.getObject(TagAttributeImpl.java:291)

at com.sun.faces.facelets.tag.TagAttributeImpl.getBoolean(TagAttributeImpl.java:107)

at com.sun.faces.facelets.tag.jstl.core.ChooseWhenHandler.isTestTrue(ChooseWhenHandler.java:46)

at com.sun.faces.facelets.tag.jstl.core.ChooseHandler.apply(ChooseHandler.java:63)

at jakarta.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:57)

at jakarta.faces.view.facelets.DelegatingMetaTagHandler.applyNextHandler(DelegatingMetaTagHandler.java:163)

at com.sun.faces.facelets.tag.faces.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:177)

at jakarta.faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:139)

at com.sun.faces.facelets.tag.ui.DefineHandler.applyDefinition(DefineHandler.java:63)

at com.sun.faces.facelets.tag.ui.CompositionHandler.apply(CompositionHandler.java:161)

at com.sun.faces.facelets.impl.DefaultFaceletContext$TemplateManager.apply(DefaultFaceletContext.java:358)

at com.sun.faces.facelets.impl.DefaultFaceletContext.includeDefinition(DefaultFaceletContext.java:329)

at com.sun.faces.facelets.tag.ui.InsertHandler.apply(InsertHandler.java:69)

at jakarta.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:57)

at com.sun.faces.facelets.tag.faces.core.FacetHandler.apply(FacetHandler.java:61)

at jakarta.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:57)

at jakarta.faces.view.facelets.DelegatingMetaTagHandler.applyNextHandler(DelegatingMetaTagHandler.java:163)

at com.sun.faces.facelets.tag.faces.ComponentTagHandlerDelegateImpl.apply(ComponentTagHandlerDelegateImpl.java:177)

at jakarta.faces.view.facelets.DelegatingMetaTagHandler.apply(DelegatingMetaTagHandler.java:139)

at jakarta.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:57)

at com.sun.faces.facelets.tag.faces.core.ViewHandler.apply(ViewHandler.java:172)

at jakarta.faces.view.facelets.CompositeFaceletHandler.apply(CompositeFaceletHandler.java:57)

at com.sun.faces.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:52)

at com.sun.faces.facelets.compiler.EncodingHandler.apply(EncodingHandler.java:46)

at com.sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:265)

at com.sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:318)

at com.sun.faces.facelets.impl.DefaultFacelet.include(DefaultFacelet.java:301)

at com.sun.faces.facelets.impl.DefaultFaceletContext.includeFacelet(DefaultFaceletContext.java:154)

at com.sun.faces.facelets.tag.ui.CompositionHandler.apply(CompositionHandler.java:129)

at com.sun.faces.facelets.compiler.NamespaceHandler.apply(NamespaceHandler.java:52)

at com.sun.faces.facelets.compiler.EncodingHandler.apply(EncodingHandler.java:46)

at com.sun.faces.facelets.impl.DefaultFacelet.apply(DefaultFacelet.java:132)

at com.sun.faces.application.view.FaceletViewHandlingStrategy.buildView(FaceletViewHandlingStrategy.java:331)

at com.sun.faces.application.view.FaceletViewHandlingStrategy.restoreView(FaceletViewHandlingStrategy.java:252)

at com.sun.faces.application.view.MultiViewHandler.restoreView(MultiViewHandler.java:126)

at jakarta.faces.application.ViewHandlerWrapper.restoreView(ViewHandlerWrapper.java:96)

at jakarta.faces.application.ViewHandlerWrapper.restoreView(ViewHandlerWrapper.java:96)

at jakarta.faces.application.ViewHandlerWrapper.restoreView(ViewHandlerWrapper.java:96)

at org.omnifaces.viewhandler.OmniViewHandler.restoreView(OmniViewHandler.java:113)

at com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:162)

at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:72)

at com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:96)

at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:131)

at jakarta.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:691)

at jakarta.faces.webapp.FacesServlet.service(FacesServlet.java:449)

at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1554)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:331)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)

at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:83)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)

at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:166)

at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)

at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)

at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)

at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)

Sherry Lake (Jun 02 2025 at 19:37):

at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:369)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239)

at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)

at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)

at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)

at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)

at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)

at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)

at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)

at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)

at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)

at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)

at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)

at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)

at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)

at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)

at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)

at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)

at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)

at java.base/java.lang.Thread.run(Thread.java:840)

Caused by: jakarta.el.ELException: /dataset.xhtml @75,75 test="#{empty DatasetPage.croissant}": java.lang.NullPointerException: Cannot invoke "edu.harvard.iq.dataverse.DatasetVersion.isPublished()" because "this.workingVersion" is null

at com.sun.faces.facelets.el.TagValueExpression.getValue(TagValueExpression.java:77)

at com.sun.faces.facelets.tag.TagAttributeImpl.getObject(TagAttributeImpl.java:289)

... 81 more

Caused by: jakarta.el.ELException: java.lang.NullPointerException: Cannot invoke "edu.harvard.iq.dataverse.DatasetVersion.isPublished()" because "this.workingVersion" is null

at jakarta.el.BeanELResolver.getValue(BeanELResolver.java:351)

at com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:139)

at com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:164)

at org.glassfish.expressly.parser.AstValue.getValue(AstValue.java:302)

at org.glassfish.expressly.parser.AstValue.getValue(AstValue.java:144)

at org.glassfish.expressly.parser.AstEmpty.getValue(AstEmpty.java:44)

at org.glassfish.expressly.ValueExpressionImpl.getValue(ValueExpressionImpl.java:138)

at org.jboss.weld.module.web.el.WeldValueExpression.getValue(WeldValueExpression.java:50)

at com.sun.faces.facelets.el.TagValueExpression.getValue(TagValueExpression.java:73)

... 82 more

Caused by: java.lang.NullPointerException: Cannot invoke "edu.harvard.iq.dataverse.DatasetVersion.isPublished()" because "this.workingVersion" is null

at edu.harvard.iq.dataverse.DatasetPage.isThisLatestReleasedVersion(DatasetPage.java:5964)

at edu.harvard.iq.dataverse.DatasetPage.getCroissant(DatasetPage.java:5984)

at jdk.internal.reflect.GeneratedMethodAccessor1211.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:569)

at jakarta.el.BeanELResolver.getValue(BeanELResolver.java:346)

... 90 more

]]

Philip Durbin 🚀 (Jun 02 2025 at 19:49):

That Croissant bug will be fixed by PR #11476 in 6.7

Sherry Lake (Jun 02 2025 at 19:50):

I just ran the index continue... and it said that only "2" datasets were to be indexed?

curl http://localhost:8080/api/admin/index/continue

I know that these are rather LARGE datasets - in number of files (over 6,000). These are the only two messages in my log when I ran the above command.

[2025-06-02T19:42:53.501+0000] [Payara 6.2025.2] [INFO] [] [edu.harvard.iq.dataverse.search.IndexBatchServiceBean] [tid: _ThreadID=216 _ThreadName=__ejb-thread-pool4] [timeMillis: 1748893373501] [levelValue: 800] [[
indexing dataset 1 of 2 (id=81905)]]

[2025-06-02T19:43:06.510+0000] [Payara 6.2025.2] [INFO] [] [edu.harvard.iq.dataverse.search.IndexBatchServiceBean] [tid: _ThreadID=216 _ThreadName=__ejb-thread-pool4] [timeMillis: 1748893386510] [levelValue: 800] [[
indexing dataset 2 of 2 (id=36552)]]

Sherry Lake (Jun 02 2025 at 19:51):

@Philip Durbin so that croissant bug isn't the problem?

I need to look for another error?

Or Can I just run the index command again?
This one: curl http://localhost:8080/api/admin/index

Philip Durbin 🚀 (Jun 02 2025 at 19:52):

Did you upgrade to Solr 9.8.0? That's part of the 6.6 release notes.

Sherry Lake (Jun 02 2025 at 19:54):

Yes, 9.8.

Sherry Lake (Jun 02 2025 at 19:55):

Another large dataset showed up (14,814 files).

Philip Durbin 🚀 (Jun 02 2025 at 19:55):

Ok, so the index is empty to start with. And now you're trying to do the initial indexing?

Sherry Lake (Jun 02 2025 at 19:56):

Yes, I ran.
curl http://localhost:8080/api/admin/index/clear

Then the index command:
curl http://localhost:8080/api/admin/index

Sherry Lake (Jun 02 2025 at 19:58):

Reason why I cleared, is because after the upgrade, I created a dataset (draft), but it did not show up in the collection. It was there, I got to it by the URL, but it didn't display in dataset listing.... so I thought the solr index needed re-indexing....

Philip Durbin 🚀 (Jun 02 2025 at 19:59):

I wonder if you should clear the index timestamps... https://guides.dataverse.org/en/6.6/admin/solr-search-index.html#clear-index-timestamps

Philip Durbin 🚀 (Jun 02 2025 at 20:00):

To be honest, I haven't done this in quite a while.

Philip Durbin 🚀 (Jun 02 2025 at 20:01):

You're saying curl http://localhost:8080/api/admin/index only showed "indexing dataset 1 of 2"? That doesn't sound right.

Sherry Lake (Jun 02 2025 at 20:02):

The "continue" endpoint:
curl http://localhost:8080/api/admin/index/continue

Philip Durbin 🚀 (Jun 02 2025 at 20:02):

Ok, that might make sense.

Philip Durbin 🚀 (Jun 02 2025 at 20:03):

Anything else in server.log after you do /clear and /index?

Philip Durbin 🚀 (Jun 02 2025 at 20:03):

Such as a complaint about a field Solr doesn't know about?

Don Sizemore (Jun 02 2025 at 20:04):

has Tim verified that the updated solrconfig.xml and schema.xml show up in place on the Solr box (and does Terraform or other restart Solr on a conf file update?)

Sherry Lake (Jun 02 2025 at 20:06):

@Don Sizemore I'll ask.

But I can see the solr mount point and checked that the solrconfig.xml there matches (diff) what I downloaded from the V6.6 installation instructions.

Sherry Lake (Jun 02 2025 at 20:07):

And the schema.xml there does have my local metadata block fields.

Sherry Lake (Jun 02 2025 at 20:10):

Before I started the "clear", Tim restarted the "core".... I assume solr.

Sherry Lake (Jun 02 2025 at 20:13):

The problem I was trying to solve was.... a new dataset (created after upgrade) was not showing in the listing.

Philip Durbin 🚀 (Jun 02 2025 at 20:15):

Right, I think you hit #11442 which is fixed by #11476. If you're seeing this a lot you could disable the Croissant exporter until the fix ships in 6.7.

Philip Durbin 🚀 (Jun 02 2025 at 20:16):

That new dataset... what if you index it? Does it show up in the listing?

Philip Durbin 🚀 (Jun 02 2025 at 20:16):

Or you could edit its metadata to force indexing of just that dataset.

Sherry Lake (Jun 02 2025 at 20:20):

Ok, editing that newly creating dataset.... now it shows up - to me - it's draft.

But I still don't see the other datasets (except the 5 that have appeared "slowly").

Philip Durbin 🚀 (Jun 02 2025 at 20:22):

Ok, so you're having two problems:

some (or all?) new datasets are not being indexed because of the Croissant bug, fixed in #11476
when doing an "index all", lots of datasets haven't been indexed yet (we need to see what's going on in server.log to troubleshoot)

Sherry Lake (Jun 02 2025 at 20:26):

Let's just get my dataset listings back:

Should I remove the timestamps?
https://guides.dataverse.org/en/6.6/admin/solr-search-index.html#clear-index-timestamps

And try to re-run index OR just re-run index?
curl http://localhost:8080/api/admin/index

Then I'll see what is in my log file. Nothing happening now in my log file.

Philip Durbin 🚀 (Jun 02 2025 at 20:28):

I guess I'd suggest

curl http://localhost:8080/api/admin/index/clear

followed by

curl http://localhost:8080/api/admin/index

Philip Durbin 🚀 (Jun 02 2025 at 20:29):

Because you're starting from basically zero anyway (only 5 datasets indexed).

Sherry Lake (Jun 02 2025 at 20:34):

It is taking a long time to clear:
curl http://localhost:8080/api/admin/index/clear

[2025-06-02T20:29:45.734+0000] [Payara 6.2025.2] [INFO] [] [edu.harvard.iq.dataverse.search.SolrIndexServiceBean] [tid: _ThreadID=75 _ThreadName=http-thread-pool::http-listener-1(3)] [timeMillis: 1748896185734] [levelValue: 800] [[

attempting to delete all Solr documents before a complete re-index]]

Nothing else in the log... and prompt has not come back.

Sherry Lake (Jun 02 2025 at 20:35):

Ok.... now these.... but is this my "clear index" or something else:

[2025-06-02T20:34:45.686+0000] [Payara 6.2025.2] [INFO] [] [edu.harvard.iq.dataverse.search.SearchServiceBean] [tid: _ThreadID=92 _ThreadName=http-thread-pool::jk-connector(3)] [timeMillis: 1748896485686] [levelValue: 800] [[

Search Syntax Error: Error from server at http://dataverse-solr-production.private.production:8080/solr/collection1/select?q=*&sort=dateSort+desc&fl=*%2Cscore&qt=%2Fselect&facet=true&facet.query=*&facet.field=dvObjectType&facet.field=metadata_type_ss&facet.field=datasetType&facet.field=dvCategory&facet.field=metadataSource&facet.field=publicationDate&facet.field=publicationStatus&facet.field=license&facet.field=authorName_ss&facet.field=subject_ss&facet.field=authorAffiliation_ss&facet.field=fileTypeGroupFacet&facet.field=fileTag&facet.field=fileAccess&fq=subtreePaths%3A%22%2F1%2F221%22&fq=authorAffiliation_ss%3A%22UVA+School+of+Architecture%22&fq=authorName_ss%3A%22Wu%2C+Junyi%22&fq=authorName_ss%3A%22Wrobel%2C+Elena%22&fq=authorAffiliation_ss%3A%22Interdisciplinary+Program+in+Archaeology%2C+UVA+Arts+&fq=dvObjectType%3A%28dataverses+OR+datasets%29&fq=%7B%21join+from%3DdefinitionPointDocId+to%3Did+v%3D%24q1%7D&hl=true&hl.snippets=1&hl.requireFieldMatch=true&hl.fragsize=320&hl.simple.pre=%3Cspan+class%3D%22search-term-match%22%3E&hl.simple.post=%3C%2Fspan%3E&hl.fl=resolution.Temporal&hl.fl=3dLightingSetup&hl.fl=distributionDate&hl.fl=discipline&hl.fl=contributorName&hl.fl=coverage.Temporal&hl.fl=datasetContactAffiliation&hl.fl=journalArticleType&hl.fl=datasetLevelErrorNotes&hl.fl=state&hl.fl=otherIdValue&hl.fl=dateOfCollectionEnd&hl.fl=studyAssayOrganism&hl.fl=3dEquipment&hl.fl=seriesName&hl.fl=coverage.Spectral.Bandpass&hl.fl=producerURL&hl.fl=redshiftType&hl.fl=astroFacility&hl.fl=coverage.Depth&hl.fl=3d3DTechnique&hl.fl=authorName&hl.fl=timeMethod&hl.fl=depositor&hl.fl=otherDataAppraisal&hl.fl=coverage.Spectral.Wavelength&hl.fl=CADFormat&hl.fl=actionsToMinimizeLoss&hl.fl=3dAltText&hl.fl=literalQuestion&hl.fl=distributorName&hl.fl=journalVolume&hl.fl=topicClassValue&hl.fl=distributorAbbreviation&hl.fl=publicationRelationType&hl.fl=coverage.Polarization&hl.fl=studyFactorType&hl.fl=coverage.Spectral.CentralWavelength&hl.fl=keywordTermURI&hl.fl=3dWeight&hl.fl=distributor&hl.fl=dsPublicationDate&hl.fl=topicClassification&hl.fl=studyOtherDesignType&hl.fl=externalContrib&hl.fl=series&hl.fl=dateOfDeposit&hl.fl=coverage.Temporal.StartTime&hl.fl=frequencyOfDataCollection&hl.fl=socialScienceNotes&hl.fl=country&hl.fl=forumLink&hl.fl=studyAssayPlatform&hl.fl=interviewInstructions&hl.fl=language&hl.fl=keywordVocabulary&hl.fl=coverage.ObjectCount&hl.fl=objectType&hl.fl=unitOfAnalysis&hl.fl=keyword&hl.fl=relatedMaterial&hl.fl=variableUniverse&hl.fl=targetSampleActualSize&hl.fl=journalIssue&hl.fl=publicationIDNumber&hl.fl=coverage.ObjectDensity&hl.fl=dataCollector&hl.fl=grantNumberAgency&hl.fl=3dExportedFilePolygonCount&hl.fl=name&hl.fl=fabTime&hl.fl=contributorType&hl.fl=notesText&hl.fl=studyDesignType&hl.fl=geographicCoverage&hl.fl=3dWidth&hl.fl=description&hl.fl=resolution.Spatial&hl.fl=title&hl.fl=studyOtherFactorType&hl.fl=3dObjectDimensions&hl.fl=productionDate&hl.fl=otherGeographicCoverage&hl.fl=kindOfData&hl.fl=socialScienceNotesSubject&hl.fl=variableName&hl.fl=studyAssayTechnologyType&hl.fl=producerName&hl.fl=targetSampleSizeFormula&hl.fl=samplingErrorEstimates&hl.fl=datasetContactEmail&hl.fl=targetSampleSize&hl.fl=disciplines&hl.fl=universe&hl.fl=resolution.Redshift&hl.fl=grantNumberValue&hl.fl=grantNumber&hl.fl=cleaningOperations&hl.fl=fileType&hl.fl=dateOfCollectionStart&hl.fl=software&hl.fl=3dUnit&hl.fl=responseRate&hl.fl=geographicBoundingBox&hl.fl=coverage.Redshift.MaximumValue&hl.fl=bigIdea&hl.fl=remixObjects&hl.fl=affiliation_ss&hl.fl=datasetContactName&hl.fl=studyAssayOtherPlatform&hl.fl=gradeLevel&hl.fl=dsDescriptionDate&hl.fl=socialScienceNotesText&hl.fl=producerLogoURL&hl.fl=consumableCost&hl.fl=timePeriodCoveredStart&hl.fl=socialScienceNotesType&hl.fl=publicationIDType&hl.fl=alternativeURL&hl.fl=3dMasterFilePolygonCount&hl.fl=fileTypeDisplay&hl.fl=authorAffiliation&hl.fl=astroType&hl.fl=producerAffiliation&hl.fl=subDiscipline&hl.fl=subtitle&hl.fl=authorIdentifierScheme&hl.fl=softwareVersion&hl.fl=coverage.SkyFraction&hl.fl=city&hl.fl=dsDescriptionValue&hl.fl=characteristicOfSources&hl.fl=coverage.Temporal.StopTime&hl.fl=studyAssayOtherOrganism&hl.fl=alternativeTitle&hl.fl=topicClassVocabURI&hl.fl=journalVolumeIssue&hl.fl=externalAgency&hl.fl=distributorURL&hl.fl=reusableCost&hl.fl=externalIdValue&hl.fl=coverage.Spatial&hl.fl=journalPubDate&hl.fl=distributorAffiliation&hl.fl=assemTime&hl.fl=fileNameWithoutExtension&hl.fl=sampleLearningGoals&hl.fl=topicClassVocab&hl.fl=originOfSources&hl.fl=resolution.Spectral&hl.fl=fabEquipment&hl.fl=variableNotes&hl.fl=westLongitude&hl.fl=dsPersistentId&hl.fl=contentStandards&hl.fl=3dHeight&hl.fl=dsDescription&hl.fl=astroObject&hl.fl=coverage.Spectral.MinimumWavelength&hl.fl=variableLabel&hl.fl=timePeriodCovered&hl.fl=studyAssayOtherTechnologyType&hl.fl=subject&hl.fl=studyAssayOtherMeasurmentType&hl.fl=accessToSources&hl.fl=controlOperations&hl.fl=samplingProcedure&hl.fl=otherReferences&hl.fl=coverage.Redshift.MinimumValue&hl.fl=eastLongitude&hl.fl=producerAbbreviation&hl.fl=publication&hl.fl=postQuestion&hl.fl=fileTags&hl.fl=dataSources&hl.fl=distributorLogoURL&hl.fl=filePersistentId&hl.fl=otherIdAgency&hl.fl=author&hl.fl=publicationCitation&hl.fl=productionPlace&hl.fl=weighting&hl.fl=softwareName&hl.fl=3dExportedFileFormat&hl.fl=producer&hl.fl=publicationURL&hl.fl=keywordValue&hl.fl=3dHandling&hl.fl=otherUseObject&hl.fl=keywordVocabularyURI&hl.fl=contributor&hl.fl=northLatitude&hl.fl=collectorTraining&hl.fl=studyAssayCellType&hl.fl=coverage.RedshiftValue&hl.fl=3dMaterialComposition&hl.fl=timePeriodCoveredEnd&hl.fl=southLatitude&hl.fl=dataCollectionSituation&hl.fl=otherId&hl.fl=dateOfCollection&hl.fl=collectionMode&hl.fl=deviationsFromSampleDesign&hl.fl=coverage.Spectral.MaximumWavelength&hl.fl=3dLength&hl.fl=authorIdentifier&hl.fl=relatedDatasets&hl.fl=datasetContact&hl.fl=researchInstrument&hl.fl=astroInstrument&hl.fl=seriesInformation&hl.fl=studyAssayMeasurementType&hl.fl=geographicUnit&q1=discoverableBy%3A%28group_public+OR+group_builtIn%2Fall-users%29&start=0&rows=10&wt=javabin&version=2: org.apache.solr.search.SyntaxError: Cannot parse 'authorAffiliation_ss:"Interdisciplinary Program in Archaeology, UVA Arts ': Lexical error at line 1, column 74. Encountered: <EOF> after prefix "\"Interdisciplinary Program in Archaeology, UVA Arts " (in lexical state 3)]]

Sherry Lake (Jun 02 2025 at 21:05):

Things are back......

I think one LARGE dataset got stuck.
This last one came through the log 20 min. ago, which was the end of indexing after the upgrade (only took 4 hours!!!!).

[2025-06-02T20:44:31.092+0000] [Payara 6.2025.2] [INFO] [] [edu.harvard.iq.dataverse.search.IndexBatchServiceBean] [tid: _ThreadID=228 _ThreadName=__ejb-thread-pool16] [timeMillis: 1748897071092] [levelValue: 800] [[
indexing dataset 835 of 835 (id=36552)]]

[2025-06-02T20:44:47.659+0000] [Payara 6.2025.2] [INFO] [] [edu.harvard.iq.dataverse.search.IndexBatchServiceBean] [tid: _ThreadID=228 _ThreadName=__ejb-thread-pool16] [timeMillis: 1748897087659] [levelValue: 800] [[
done iterating through all datasets]]

Sherry Lake (Jun 02 2025 at 22:51):

(deleted)

Sherry Lake (Jun 02 2025 at 22:52):

we are back!

I would like to suggest that an index clear be recommended before a re index - in the installation notes.

The reindex was “quick” after clearing the index versus just reindexing in place. An hour compared to 4+ hours.

Thanks for the advice.

Philip Durbin 🚀 (Jun 02 2025 at 22:53):

@Sherry Lake I'm glad you're back! Good job.

Notification Bot (Jun 03 2025 at 14:34):

Sherry Lake has marked this topic as resolved.

Last updated: Oct 30 2025 at 06:21 UTC