Re: Using a real mirroring tool...

[Charles Wilson <cwilson at ece dot gatech dot edu>]

A few notes:

Using rsync does not help the following problem:  I have four favorite 
mirrors, plus a "private" setup-compatible website from my friend Bob. 
setup.exe will show me a merged view of the contents of all of those 
sites, insuring that I get the latest version of a package wherever it 
is found.  By merging in the "private" site, setup enables me to get my 
friend Bob's latest version of "King of Frobozz, Tyrant" game using the 
same setup tool/view, along with the latest apache package or binutils 
package from the "official" cygwin mirrors.

rsync can't merge.

I could rsync each of the four mirrors down to my machine into separate 
directories (assuming the mirrors all had anonymous rsync servers), and 
rsync Bob's private website (assuming his webhost provided rsync -- 
hah!).  But even then, setup can't merge multiple local directories, at 
least not right now.

So, we still need some variant of setup.exe that can merge multiple 
setup-compatible sites, and "mirror" only the latest stuff into a single 
  local "release" tree / setup.ini.

Also, rsync does some stuff we don't really care about: since most of 
the files are compressed, rsync's sophisticated "download changed 
chunks" algorithm is pretty useless.  In compressed files, the first 
byte that is different in the uncompressed file will cause a cascade of 
changes throughout the remainder of the compressed file.  So: 1 byte 
diff in uncompressed file == entire compressed file is different.

OTOH, since we typically (never?) upload a modified version of a given 
file USING THE SAME NAME, we don't really care too much about this. 
"Our" mirroring tool can just use the md5 hashes on the mirror and 
locally to determine the rare instance when a given file has had its 
contents changed -- but then we'd download the ENTIRE file, not "changed 
chunks".  (However, if this ever happens, isn't it a sign that something 
went wrong somewhere?)

Also, there's really no need to use rsync's compression engine, since 
most of the data to be transferred has already been compressed.

Finally, the local "setup.ini" will never(?) match the setup.ini at any 
one of the mirrors or private sites; it will be a merger of them.  So, 
rsync can't do that.

So, if someone cares enough to do the work, there is still a need for 
the "additional tool" that uses the setup.exe codebase to do 
"cygwin-specific distributed mirroring".  See my earlier rant.

But, the rsync method should enable folks with slow connections to limp 
along for a while...without trying to turn setup.exe ITSELF into a 
mirror tool.

--Chuck


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Bug reporting:         http://cygwin.com/bugs.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/