RDS looks like gzip file · troubleshooting

Hello ,
(We are running 6.10.1, though this dataset was ingested when we were using 6.6)

We have this dataset that has 4 RDS files in it. 3 were converted to tab files and the 4th was not because it looks like a gzip file. In looking at the files, I see the all have the same bytes in the header indicating they are RDS compressed files.
I can't tell what is different about the files that one would be identified as GZIP while the rest are identified as RDS. I'm not familiar with R and only have done some basic troubleshooting.

$ hexdump -n 2 Vote_Data.RDS
0000000 8b1f
0000002
$ hexdump -n 2 College_Debt_Data.RDS
0000000 8b1f
0000002

Bethany Seeger (Jul 23 2026 at 18:03):

Note: I did try taking these files from our production server and importing to our test. I got the same results.

Philip Durbin 🚀 (Jul 23 2026 at 18:07):

% file College_Debt_Data.RDS Vote_Data.RDS
College_Debt_Data.RDS: gzip compressed data, from HPFS filesystem (OS/2, NT), original size modulo 2^32 117202
Vote_Data.RDS:         gzip compressed data, from HPFS filesystem (OS/2, NT), original size modulo 2^32 251482

Philip Durbin 🚀 (Jul 23 2026 at 18:08):

@Bethany Seeger the file that didn't ingest is so small I would suggest creating an issue and uploading it there. You might have to put it in a .zip file.

Bethany Seeger (Jul 23 2026 at 18:09):

Thanks, @Philip Durbin 🚀 . Are you suggesting the system might have an issue around really small RDS compressed files? Perhaps that that is the issue? interesting!

Philip Durbin 🚀 (Jul 23 2026 at 18:10):

Philip Durbin 🚀 (Jul 23 2026 at 18:12):

Philip Durbin 🚀 (Jul 23 2026 at 18:13):

I just meant I think there's a size limit to files you can attach to a GitHub issue. That file that won't ingest is tiny so it should upload fine.

Philip Durbin 🚀 (Jul 23 2026 at 18:16):

Bethany Seeger (Jul 23 2026 at 18:16):

Ohhhhh... okay. Gottcha. Thanks for looking at the files. I have limited working knowledge with R and via file/hexdump I couldn't see any obvious difference in the files. Perhaps there is a difference inside the object structure that failed.

Bethany Seeger (Jul 23 2026 at 18:17):

So, I just realized I didn't look at the logs. I deleted and reuploaded the file and saw this:

Philip Durbin 🚀 (Jul 23 2026 at 18:17):

Bethany Seeger (Jul 23 2026 at 18:33):

I just downloaded RStudio and was able to load() the file successfully in there.

Bethany Seeger (Jul 23 2026 at 18:33):

Bethany Seeger (Jul 23 2026 at 18:34):

Bethany Seeger (Jul 23 2026 at 18:42):

I'll create a ticket, but I half suspect there might be an issue in the file that's not immediately obvious.

Philip Durbin 🚀 (Jul 23 2026 at 18:43):

Bethany Seeger (Jul 23 2026 at 18:44):

Here's more of the stack, which probably contains the most relevant Dataverse info (let me know if you want the full stack)

[2026-07-23T18:16:23.359+0000] [Payara 7.2026.2] [SEVERE] [] [] [tid: _ThreadID=103 _ThreadName=http-thread-pool::jk-connector(2)] [timeMillis: 1784830583359] [levelValue: 1000] [[
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:266)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:175)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:149)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:123)
at edu.harvard.iq.dataverse.ingest.IngestableDataChecker.testRDAformat(IngestableDataChecker.java:593)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at edu.harvard.iq.dataverse.ingest.IngestableDataChecker.detectTabularDataFormat(IngestableDataChecker.java:669)
at edu.harvard.iq.dataverse.ingest.IngestableDataChecker.detectTabularDataFormat(IngestableDataChecker.java:639)
at edu.harvard.iq.dataverse.util.FileUtil.determineFileType(FileUtil.java:458)
at edu.harvard.iq.dataverse.engine.command.impl.CreateNewDataFilesCommand.execute(CreateNewDataFilesCommand.java:218)
at edu.harvard.iq.dataverse.engine.command.impl.CreateNewDataFilesCommand.execute(CreateNewDataFilesCommand.java:69)
at edu.harvard.iq.dataverse.EjbDataverseEngineInner.submit(EjbDataverseEngineInner.java:36)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.glassfish.exousia.AuthorizationService.lambda$invokeBeanMethod$2(AuthorizationService.java:498)
at org.glassfish.exousia.AuthorizationService.runInScope(AuthorizationService.java:582)
at org.glassfish.exousia.AuthorizationService.invokeBeanMethod(AuthorizationService.java:498)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:272)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4716)

Philip Durbin 🚀 (Jul 23 2026 at 18:46):

Philip Durbin 🚀 (Jul 23 2026 at 18:47):

Bethany Seeger (Jul 23 2026 at 18:49):

For 6.10.1 upload - yes. And I don't have access to the actual (pre-Dataverse) original files, so I downloaded the originals from our prod.

Yeah, seems like that spot makes sense. AI says that if the GZIPOutputStream or ObjectOutputStream wasn't closed on the write, it could cause this error reading it in. So it's possible it's the file.

Bethany Seeger (Jul 23 2026 at 18:50):

I have to step away for a while, but I might try writing that file again (now that I have it open) and see how that goes.

Philip Durbin 🚀 (Jul 23 2026 at 18:51):

Sure, maybe a workaround it to rewrite the file. Still smells like a bug to me though. :lady_beetle:

Bethany Seeger (Jul 24 2026 at 14:20):

Hi Philip, I tried to resave the file, but it still shows up as a gzip archive. If it is a format issue, I don't know enough R to fix it. I'm seeing if we can ask the depositor to check.
I'll create a ticket though, as it could be something else, or maybe Dataverse could handle it better if it is format.

Philip Durbin 🚀 (Jul 24 2026 at 14:23):

Not to be nosy, but why is RData/RDS (application/x-rlang-transport) the preferred format for this researcher? Richer metadata?

Philip Durbin 🚀 (Jul 24 2026 at 14:27):

Philip Durbin 🚀 (Jul 24 2026 at 14:28):

Philip Durbin 🚀 (Jul 24 2026 at 14:29):

          "contentType": "text/tab-separated-values",
          "friendlyType": "Tab-Delimited",
          "filesize": 161484,
          "description": "Vote_Data; ISPS number D228F05; Published 4/23/2026; Source: Authors; Created with: R ",
          "categories": [
            "Data",
            "Survey/interview (e.g., ANES)"
          ],
          "storageIdentifier": "s3://yul-dv-prd:19dc0231be9-8b8496ecf6a8",
          "originalFileFormat": "application/x-rlang-transport",
          "originalFormatLabel": "R Data",
          "originalFileSize": 23293,
          "originalFileName": "Vote_Data.RDS",

Bethany Seeger (Jul 24 2026 at 14:30):

Philip Durbin 🚀 (Jul 24 2026 at 14:30):

Bethany Seeger (Jul 24 2026 at 14:32):

Hmmm..... I think we turned Rdata ingest off because of a security recommendation.

Philip Durbin 🚀 (Jul 24 2026 at 14:32):

Bethany Seeger (Jul 24 2026 at 14:33):

This might be a stupid question, but how is Rdata different from RDS? I know RDS is an object (compressed or not), is RData as well? nm, I'll google it! :)

Philip Durbin 🚀 (Jul 24 2026 at 14:34):

Philip Durbin 🚀 (Jul 24 2026 at 14:35):

Bryan Gee (Jul 27 2026 at 13:32):

Just chiming in to say that I think the issue of .rds files being read as .gzip files is not a Dataverse-specific issue; I remember this issue coming up at Dryad more than 3 years ago now (and I don't think they came up with a workaround it yet) - RDS files are compressed using GZIP compression, and this causes problems with how automated systems read them. The gist of RDS vs RData is mainly in how many objects can be in each one and how they interface with the workspace when loaded into R (this link does a good job of summarizing: https://www.datanovia.com/learn/programming/importing-exporting-data/save-rds-rdata); they are preferred by some to more open formats because they support more dimensionality than a standard CSV or Excel file (sort of like how some people prefer Excel to CSV because they can have multiple sheets, formulas, macros, etc.) and preserve the exact original structure, which can be lost when converting. When shared in tandem with the R code, they probably improve reproducibility, but they are not particularly interoperable otherwise.

Philip Durbin 🚀 (Jul 27 2026 at 13:33):

Philip Durbin 🚀 (Jul 27 2026 at 13:34):

I mentioned the new issue Bethany opened at #12558 to @Leo Andreev and he plans to comment on it when has time.

Philip Durbin 🚀 (Jul 27 2026 at 13:36):

Bethany Seeger (Jul 27 2026 at 13:50):

Thanks, @Bryan Gee - My take away is that whether they show up as a GZIP or RDS file in dataverse doesn't matter - it will still operate the same when the user plugs it into their R setup. In your experience, is that true?

Bryan Gee (Jul 27 2026 at 14:06):

In theory, it should work. I haven't had to interface with one of these in our installation in a while (though I can probably find one to do some testing with). It might require requiring the download to use the original file format instead of the converted .tab if it did that, and there could be behavioral differences if downloading a single .rds file versus downloading everything in one .zip file - I vaguely recall that the issue in Dryad's system only happened when you downloaded an individual .rds file. However, I think what happened in that case is that it downloaded the single file with a .gz extension, so someone would have to know to rename the extension to .rds and ignore whatever warnings the OS threw about doing that

Bryan Gee (Jul 27 2026 at 14:09):

Although the metadata says it's a GZIP file, it downloads with the correct extension, and I could read it into R correctly

Bethany Seeger (Jul 27 2026 at 14:44):

Leo Andreev (Jul 28 2026 at 16:46):

Strictly speaking, Dataverse isn't lying - RDS is a gzip file (just like RData; both are compressed using gz algorithm). ... But, this is of course not ideal, that the type is not properly recognized.
I will add a some comments to the issue 12558
But I just want to say that I am very much relieved to hear that you have disabled R ingest on your instance(s). That was a serious issue (that would be present with RDS as well).
So, a short/tl;dr version: two separate things: a) We DO want to recognize RDS files, as mime type application/x-rds, with some meaningful human-friendly label; instead of the confusing "gzip". But b) No, we do not want to ingest it, unless/until somebody re-engineers the obsolete/deprecated/abandoned R framework in Dataverse. If it is important to have the data frame in that file ingested as tabular data, the recommended workaround should be to export it as CSV in R; then ingest the .csv file. There is virtually nothing useful that Dataverse knows how to learn from an R data frame, that would not be present in the CSV export.

Philip Durbin 🚀 (Jul 28 2026 at 17:10):

@Leo Andreev thanks for your comments. I guess one of my takeaways is that researchers really are using the RDS format. That's the newer one, newer than .RData. And yeah, we should come up with a better way to ingest them. Ista Zahn has some ideas years ago but we can pick this up in #8526.

Leo Andreev (Jul 28 2026 at 17:18):

I may have missed something, but based on what I read, it didn't appear to be any newer than RData. Or at least it was already around when R support was implemented in DVN/Dataverse back in the middle ages. Looked more like another instance of ancient legacy, the rationale for which time has forgotten, that the type detection for it was never implemented, and that it was not covered by the R ingest plugin.
If nothing else, ingesting it would actually be easier/more straightforward than RData - since there is less flexibility by design; only one data frame is guaranteed to be present, etc.

Philip Durbin 🚀 (Jul 28 2026 at 17:20):

Leo Andreev (Jul 28 2026 at 17:30):

Nothing is going to force me to watch youtube videos about R :smile:. But sure, the docs said "has been there internally all along; made exportable in the early 2010s". It was not "new" during the time when R support in Dataverse was actively worked on, was my main point. So,

...

Philip Durbin 🚀 (Jul 28 2026 at 17:30):

Bryan Gee (Jul 28 2026 at 17:33):

In my experience, RDS / RData files tend to be heavily used mainly by folks in the biological sciences. I would point out that RDS files can only contain one object compared to RData, but that does not mean that it is a single dataframe that can be directly converted to a CSV without loss of information, which is why a straight RDS to CSV conversion (either manually or by the system ingest) is not always viable (but sometimes it is, that might explain the discrepancies that Bethany was seeing among her four files). This is an example of everything in a Seurat object that I was looking at yesterday (from https://dataverse.tdl.org/file.xhtml?fileId=593146&version=1.0). Seurat is an extremely popular bioinformatics tool that is default exported in RDS, although I think there are ways to convert to others like .h5 derivatives

Leo Andreev (Jul 28 2026 at 17:54):

I am sure that it is true - that R can save a lot of useful information in either an .RDS or .RData formats, that will be lost when exporting to CSV. But please note that I was talking about tabular data ingest in Dataverse specifically:

Again, we are potentially talking about 2 different things:
a) being able to upload an .RDS file and add it to a dataset, and have it shown on the page with a meaningful label describing it as an R file and downloadable with a valid mime type; instead of the confusing "gzip". We should be able to fix this part easily.
b) Tabular data ingest, or just "ingest", as a feature in Dataverse - the process of parsing data files and extracting the metadata describing the variables and generating summary stats, etc. This was implemented for RData, but has never worked particularly well, compared to other rich formats, like Stata and SPSS. Among other things, it really does not produce any extra metadata to speak of, compared to the corresponding csv export. As I mentioned earlier, that code has been de-facto abandoned and is no longer safe to use.
Somebody may want to take over and revive that subsystem one day. Unfortunately, I do not see the main dev. team having any capacity to work on that in any foreseeable future.

Leo Andreev (Jul 28 2026 at 18:02):

With ingest in general, I want to live to see the day when it is moved out of the monolith Dataverse application and made into a standalone service. Where it would be easy for the community instances to add their own ingest implementations for whatever formats that may be important to them; with drop-in ingest plugins for specific formats that can be published and offered to other instances... i.e., ~ like metadata export plugins now.

Stream: troubleshooting

Topic: RDS looks like gzip file

Bethany Seeger (Jul 23 2026 at 18:02):

Bethany Seeger (Jul 23 2026 at 18:03):

Philip Durbin 🚀 (Jul 23 2026 at 18:07):

Philip Durbin 🚀 (Jul 23 2026 at 18:08):

Bethany Seeger (Jul 23 2026 at 18:09):

Philip Durbin 🚀 (Jul 23 2026 at 18:10):

Philip Durbin 🚀 (Jul 23 2026 at 18:12):

Philip Durbin 🚀 (Jul 23 2026 at 18:13):

Philip Durbin 🚀 (Jul 23 2026 at 18:16):

Bethany Seeger (Jul 23 2026 at 18:16):

Bethany Seeger (Jul 23 2026 at 18:17):

Philip Durbin 🚀 (Jul 23 2026 at 18:17):

Bethany Seeger (Jul 23 2026 at 18:33):

Bethany Seeger (Jul 23 2026 at 18:33):

Bethany Seeger (Jul 23 2026 at 18:34):

Bethany Seeger (Jul 23 2026 at 18:42):

Philip Durbin 🚀 (Jul 23 2026 at 18:43):

Bethany Seeger (Jul 23 2026 at 18:44):

Philip Durbin 🚀 (Jul 23 2026 at 18:46):

Philip Durbin 🚀 (Jul 23 2026 at 18:47):

Philip Durbin 🚀 (Jul 23 2026 at 18:47):

Philip Durbin 🚀 (Jul 23 2026 at 18:47):

Bethany Seeger (Jul 23 2026 at 18:49):

Bethany Seeger (Jul 23 2026 at 18:50):

Philip Durbin 🚀 (Jul 23 2026 at 18:51):

Bethany Seeger (Jul 24 2026 at 14:20):

Philip Durbin 🚀 (Jul 24 2026 at 14:23):

Philip Durbin 🚀 (Jul 24 2026 at 14:23):

Philip Durbin 🚀 (Jul 24 2026 at 14:27):

Philip Durbin 🚀 (Jul 24 2026 at 14:28):

Philip Durbin 🚀 (Jul 24 2026 at 14:29):

Bethany Seeger (Jul 24 2026 at 14:30):

Philip Durbin 🚀 (Jul 24 2026 at 14:30):

Bethany Seeger (Jul 24 2026 at 14:32):

Philip Durbin 🚀 (Jul 24 2026 at 14:32):

Bethany Seeger (Jul 24 2026 at 14:33):

Philip Durbin 🚀 (Jul 24 2026 at 14:34):

Philip Durbin 🚀 (Jul 24 2026 at 14:35):

Bryan Gee (Jul 27 2026 at 13:32):

Philip Durbin 🚀 (Jul 27 2026 at 13:33):

Philip Durbin 🚀 (Jul 27 2026 at 13:34):

Philip Durbin 🚀 (Jul 27 2026 at 13:36):

Bethany Seeger (Jul 27 2026 at 13:50):

Bryan Gee (Jul 27 2026 at 14:06):

Bryan Gee (Jul 27 2026 at 14:09):

Bethany Seeger (Jul 27 2026 at 14:44):

Leo Andreev (Jul 28 2026 at 16:46):

Philip Durbin 🚀 (Jul 28 2026 at 17:10):

Leo Andreev (Jul 28 2026 at 17:18):

Philip Durbin 🚀 (Jul 28 2026 at 17:20):

Leo Andreev (Jul 28 2026 at 17:30):

Philip Durbin 🚀 (Jul 28 2026 at 17:30):

Bryan Gee (Jul 28 2026 at 17:33):

Leo Andreev (Jul 28 2026 at 17:54):

Leo Andreev (Jul 28 2026 at 18:02):

Leo Andreev (Jul 31 2026 at 23:53):