parquet - How to fix java.nio.file.InvalidPathException: Illegal char <:> on windows pc - Stack Overflow

admin管理员组

文章数量:1122826

java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:/C:/Users/john/AppData/Local/Temp/royalty-export1794031690892090231.parquet
    at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) ~[na:na]
    at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) ~[na:na]
    at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) ~[na:na]
    at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) ~[na:na]
    at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:232) ~[na:na]
    at java.base/java.nio.file.Path.of(Path.java:147) ~[na:na]
    at java.base/java.nio.file.Paths.get(Paths.java:69) ~[na:na]
    at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:392) ~[parquet-hadoop-1.14.2.jar:1.14.2]
    at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:918) ~[parquet-hadoop-1.14.2.jar:1.14.2]
    at com.usemogul.core.reportingestion.ingest.ClickHouseReportRecordIngestionService.ingest(ClickHouseReportRecordIngestionService.java:132) ~[classes/:na]

I am trying to run a codebase that I usually run on macos, on a windows machine. I dont have this problem on the mac but on windows I keep getting this error. Not sure how to fix.

File exportFile = File.createTempFile("royalty-export", ".parquet");
        OutputFile outputFile = HadoopOutputFile.fromPath(new org.apache.hadoop.fs.Path(exportFile.getPath()), new Configuration());

        try (ReportIterator iterator = processor.iterator(report, file);
             ParquetWriter<GenericData.Record> writer = AvroParquetWriter
                     .<GenericData.Record>builder(outputFile)
                     .withSchema(SCHEMA)
                     .withValidation(true)
                     .withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
                     .build()) {

I tried looking online for solution but I couldnt find one that fit my current scenario.

java.nio.file.InvalidPathException: Illegal char <:> at index 4: file:/C:/Users/john/AppData/Local/Temp/royalty-export1794031690892090231.parquet
    at java.base/sun.nio.fs.WindowsPathParser.normalize(WindowsPathParser.java:182) ~[na:na]
    at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:153) ~[na:na]
    at java.base/sun.nio.fs.WindowsPathParser.parse(WindowsPathParser.java:77) ~[na:na]
    at java.base/sun.nio.fs.WindowsPath.parse(WindowsPath.java:92) ~[na:na]
    at java.base/sun.nio.fs.WindowsFileSystem.getPath(WindowsFileSystem.java:232) ~[na:na]
    at java.base/java.nio.file.Path.of(Path.java:147) ~[na:na]
    at java.base/java.nio.file.Paths.get(Paths.java:69) ~[na:na]
    at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:392) ~[parquet-hadoop-1.14.2.jar:1.14.2]
    at org.apache.parquet.hadoop.ParquetWriter$Builder.build(ParquetWriter.java:918) ~[parquet-hadoop-1.14.2.jar:1.14.2]
    at com.usemogul.core.reportingestion.ingest.ClickHouseReportRecordIngestionService.ingest(ClickHouseReportRecordIngestionService.java:132) ~[classes/:na]

I am trying to run a codebase that I usually run on macos, on a windows machine. I dont have this problem on the mac but on windows I keep getting this error. Not sure how to fix.

File exportFile = File.createTempFile("royalty-export", ".parquet");
        OutputFile outputFile = HadoopOutputFile.fromPath(new org.apache.hadoop.fs.Path(exportFile.getPath()), new Configuration());

        try (ReportIterator iterator = processor.iterator(report, file);
             ParquetWriter<GenericData.Record> writer = AvroParquetWriter
                     .<GenericData.Record>builder(outputFile)
                     .withSchema(SCHEMA)
                     .withValidation(true)
                     .withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
                     .build()) {

I tried looking online for solution but I couldnt find one that fit my current scenario.

• java
• parquet

Share Improve this question Follow edited Nov 23, 2024 at 15:34 OneCricketeer 191k2020 gold badges141141 silver badges267267 bronze badges asked Nov 23, 2024 at 3:26 John KwonJohn Kwon 111 bronze badge 2

• 1 It appears that for some reason, the app is attempting to open a (questionable!) "file:" URL as if it was a file system path. You need to do some debugging here. That is the value of exportFile? What does outputFile.getPath() return? – Stephen C Commented Nov 23, 2024 at 3:51
• 1 According to the stack trace you posted, I'm guessing that class com.usemogul.core.reportingestion.ingest.ClickHouseReportRecordIngestionService is one that you wrote. Again, according to the stack trace in your question, that class's ingest method is calling method build of class org.apache.parquet.hadoop.ParquetWriter.Builder. So did you post code of method ingest? – Abra Commented Nov 23, 2024 at 11:07

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 0

The problem lies in this: new org.apache.hadoop.fs.Path(exportFile.getPath()). The documentation says that path strings are URIs, but the result of exportFile.getPath() isn't a URI - it's a local file path. In this case it's also an absolute file path. On Linux and Mac this starts with / which makes it a valid URI without scheme. On Windows it starts with C:\ (or some other drive) which isn't a valid URI.

Use new org.apache.hadoop.fs.Path(exportFile.toURI()) instead. This creates a proper URI for the file.

本文标签： parquetHow to fix javaniofileInvalidPathException Illegal char ltgt on windows pcStack Overflow