Comments (8)
Ran a small script to rename all the top level folders in crates/
to lowercase, it seems to have resolved some of the problems.
It seems the same thing has happened with prefixes that start with numbers, like crates/3/F/fnv/fnv-x.y.z.crate
.
After renaming all directories to lowercase all problems have been fixed.
from romt.
I seem to have figured out the problem, some crates' names start with a capital letter, and these crates seem to be downloaded first, that causes the prefix paths to be created with uppercase letters as well, then other crates with lowercase names with the same prefix get downloaded into the already existing directories (windows paths seem to be non case-sensitive) causing the problems.
Renaming all prefixes to lowercase then breaks the downloading of the uppercase named crates.
from romt.
Thanks for submitting this issue. I haven't documented your use case (sharing a single mirror with both case-sensitive and case-insensitive mounts), but it is supported, with caveats.
TL/DR: If you write the crates using a case-insensitive mount, you must access the crates case-insensitively thereafter. If you want to use the same underlying storage with both case-sensitive and case-insensitive mounts, you must always use the case-sensitive mount when writing to the storage.
The easiest way to recover is to start over from the original crates.tar.gz
, this time using the Linux host with a case-sensitive mount. A case-insensitive mount of that filesystem may then be used for read-only serving if desired.
If you no longer have the original crates.tar.gz
file, you may recreate it from your offline mirror as follows:
-
Use the case-insensitive mount to access the share.
-
Start in the mirror directory.
-
Create crates.tar.gz for the entire mirror:
romt crate --start 0 --end master pack
The above steps should work even after you've renamed your prefix directories to be all lowercase, as long as you access the mirror case-insensitively. This is because Romt uses the crates.io-index
Git repository to determine each crate's official name (including case), and it retains the correct case in the paths within the crates.tar.gz
tarball (even if the on-disk crates/
tree uses a different case).
Unconditionally forcing the crate prefix to lowercase runs into a couple of difficulties. The main problem is when using nginx or some other server instead of romt serve
. Rust tooling requests packages with URLs of the form /crates/SomePackage/SomePackage-1.2.3.crate
. The README shows how to use nginx rewrite rules to calculate the prefix from the package name, but the technique doesn't allow for converting the prefix to lowercase; it just blindly extracts the first characters of the package name and uses them in the prefix. So for a mixed-case name like SomePackage
, the nginx rules can't create the prefix so/me/
in lowercase. For this reason, Romt uses case-sensitive prefixes that match the case of the package itself. This works on case-sensitive filesystems as expected, but also works if the crates have been written to a case-insensitive filesystem. In the latter case, directories that would be distinct on a case-sensitive filesystem (such as so/
and So/
) do merge together, but because subsequent accesses are case-insensitive, the crates are found correctly and may be served without correcting the case in the URL.
The other issue is one of backward compatibility with the way Romt has historically worked. Forcing crate prefixes to lowercase would cause failures for non-lowercase crate prefixes for users with existing mirrors.
A couple of years ago I requested that Cargo add the ability to use lowercase prefixes for crate downloads to someday make this issue easier: rust-lang/cargo#8267
The change was accepted, so if older Rust tooling need not be supported, in the future it would be possible to have Rust tooling use crate URLs with lowercase-only prefixes using the new {lowerprefix}
marker; however, Romt currently doesn't have a way to take advantage of this {lowerprefix}
feature.
I want to consider all of the above aspects more fully before deciding what changes to make (though the changes will certainly at least include additions to the README to explain the current limitations).
from romt.
hmm I see.
The nginx problem probable can be solved using perl.
The backwards compatibility thing can be "solved" by incrementing the major version (or minor since romt is still in 0.x).
But the repackaging and re-extracting solves my problem, though I recommend adding the case-sensitivity information to the readme.
from romt.
While considering the changes I want to make, I discovered that Windows case-insensitive shares of case-sensitive filesystems don't work as I'd originally thought.
Consider creating the following tree using Linux on a case-sensitive file system:
mkdir /m/tmp/rust
cd /m/tmp/rust
mkdir directory DIRECTORY
touch directory/crate DIRECTORY/CRATE
tree
with output:
├── directory
│ └── crate1
└── DIRECTORY
└── CRATE2
2 directories, 2 files
Now on a Windows machine accessing this same share via Samba (as m:\tmp\rust
), we can see both directory names (directory
and DIRECTORY
):
M:\tmp\rust>dir /b
directory
DIRECTORY
I'd thought this behavior carried over to contents within the subdirectories, but Windows is unable to see the contents of one of these directories, e.g.:
M:\tmp\rust>dir /s /b
M:\tmp\rust\directory
M:\tmp\rust\DIRECTORY
M:\tmp\rust\directory\crate1
M:\tmp\rust\DIRECTORY\crate1
Note how crate1
is found twice (once with each spelling of directory
/DIRECTORY
), whereas CRATE2
can't be accessed, e.g.:
M:\tmp\rust>copy DIRECTORY\CRATE2
The system cannot find the file specified.
Romt therefore currently doesn't support using the same tree of crates in both case-sensitive and case-insensitive mode simultaneously. Reading and writing exclusively in one mode or the other works. I've reopened this issue pending a fix.
from romt.
From my experiments, on the same network drive, mounted on two linux systems:
- fedora using nfs (case sensitive)
- ubuntu using cifs (case insensitive)
When creating directories with the same name but different casing from the fedora machine:
aa, aA, Aa, AA
appear as aa, AA~1, AA~2, AA~3
on the ubuntu machine.
The numbers seem to correspond to the directories' order of creation rather than something to do with the casing.
Anyway currently I'm planning to serve the crates just from the fedora machine so it works for me, but I agree that something should be figured out.
from romt.
Romt-0.4.0 now can use lowercase crate prefixes in addition to mixed-case prefixes. Lowercase is the default. Crate archives continue to use mixed-case prefixes for interoperability with older Romt, though new Romt can now use either prefix format.
See the README.rst
section on "Upgrading from Romt versions before 0.4.0" for more details.
from romt.
Closing with the assumption that Romt-0.4.0 fixes this issue.
from romt.
Related Issues (12)
- issue serving on offline laptop HOT 4
- Spurious network error; couldn't resolve host: crates.io HOT 9
- Offline mirroring of a subset of crates.io HOT 1
- faield to download rustup in WSL HOT 2
- HTTP requests take a long time to process HOT 5
- The argument --keep-going does not seem to be working. I can't seem to get past one of the crates having a 403 forbidden Error. HOT 3
- The romt cannot be downloaded normally. HOT 5
- GIT_HTTP_BACKEND_SOURCES should be overwritable HOT 3
- Crate unpack error moving from Mac to Windows 10 HOT 5
- What means ConnectTimeout('') and ReadTimeout('') ? HOT 3
- toolchain - nightly and stable mix/share targets HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from romt.