This is the mail archive of the cygwin-apps@cygwin.com mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

Re: patches to vendor source trees - discussion


Robert Collins wrote:


>>How about a script?  Call it say pristine-src or something like that. 
>>You would give it a package-version for the parameter.  Then the script
>>would apply the patch and if successful remove the patch file.  You
>>could even give it some options to allow for a backup of the current src
>>directory first.
>>
> 
> Sure. But this is all predicated on the patch being in the source dir.
> (And someone needs to write the script).
> 
> Chuck, I haven't heard from you on this bar the initial comments - and
> given the number of packages you maintain....
> 
> Also, please note: I'm not suggesting that anyone has to repackage
> existing stuff... only that new packages, and new releases of existing
> packages should follow the guidelines we establish. I think anything
> else would be madness.

You've stated several times that you're looking only at "baby steps" -- but 
I think baby steps are counterproductive here.  Imagine a sequence of 5 
baby step changes in the src packaging.  There are three (bad) choices:

1) after each baby step, force all maintainers to update their src packages 
to the new standard.  This is, as you say, "madness".

2) Enforce that after each baby step, new packages and updated packages 
must obey whatever new standard is in place at that time.  this will lead 
to a HUGE hodgepodge of src packages, each obeying different "standards" as 
the standard evolves over these baby steps.

3) don't force anybody to change.  You end up with the worst of (2) coupled 
with no real "standard" at all.

I'm not opposed to doing away with CYGWIN-PATCHES, or putting the patch 
file as a sibling to the package-src directory, or whatever -- I just think 
these changes, these baby-steps, will prove counterproductive in the long 
run.  A data point: I wanted to update my 10 dll-ized library packages so 
that they would accept the -DALL_STATIC flag as opposed to requiring 
various incantations of -DZLIB_STATIC -DPNG_STATIC ....

This took what? six months?  The update cycle for dependent packages is 
VERY long -- if you want to minimize risk.  Thus, the likelihood of (2) 
occuring is very high, IMO.  If we're gonna restructure the src packaging, 
let's define the BULK of it now, and tweak later -- instead of tweak now 
and radically restructure later, or tweak now, tweak later, and continue 
tweaking forever.  *Now* is when the new super-spiffy setup.exe is coming 
out.  Big changes are in the wind, why not this, too?

Of course, I'm not saying enforce immediate switchover of all packages NOW 
-- just define the new standard, however radical, and then switch each 
package over when it is updated in the normal course of events.

As much as Chris doesn't like it, there ARE advantages to the RPM style. 
Here's my take (I'll use zlib as my example):

/usr/src/cygwin/SRC
   *unpacked* sources go here. Packages that do not support
   VPATH or for some reason must be built within their src
   directory are also compiled here.
/usr/src/cygwin/BUILD
   "normal" packages are compiled here.
/usr/src/cygwin/ORIG
   .tgz's and primary sources (patch files and the like)
   go here.  "ORIG" is not an ideal name, but it's shorter
   than "PRIMARY_SOURCES"...
/usr/src/cygwin/SCRIPTS
   (think SPECS)
/usr/src/cygwin/INST
   (I've never liked RPM's habit of using /var/tmp/<pkg>/
   for temporary installs}
/usr/src/cygwin/PACKAGES
   This is where the newly-created .tar.bz2's and
   -src.tar.bz2's end up.
/usr/src/cygwin/DOCS
   cygwin-specific READMEs and stuff.

So, zlib-1.1.3-9-src.tar.bz2 contains:
/usr/src/cygwin/ORIG/zlib/zlib-113.tgz
   (pristine tarball, in whatever format the primary
    developers provide it.  No tarball renaming.
    /etc/setup/zlib-src.list.gz will take care of
    removing it at the proper time.  BTW, *DOES* setup.exe
    worry about files installed from -src packages?)
/usr/src/cygwin/ORIG/zlib/zlib-1.1.3-9.patch
    contains diffs and additional files to make zlib
    build properly.  Does NOT create the cgywin-specific
    README (that goes in DOCS).  Does NOT create the
    zlib-postinstall script.
/usr/src/cygwin/ORIG/zlib/zlib-1.1.3-9.postinstall
/usr/src/cygwin/ORIG/zlib/*
    other patches and semi-standard source files (like the
    lossless jpg patch)
/usr/src/cygwin/DOCS/zlib/zlib-1.1.3.README
    Goes here, not in the .patch or inside the tarball.
    Why? because these cygwin-specific READMEs are not
    usually intended for submission back to the upstream
    maintainers.  So, keep 'em out of the .patch; makes it
    easier for the cgywin maintainer to organize the patches
    for upstream submission...
    Also, this README no longer contains build instructions,
    (except "run /usr/src/cygwin/SCRIPTS/zlib-build.  Here
    are the allowed options:")
/usr/src/cygwin/SCRIPTS/zlib-build
    poor man's rpm+spec.  A shell script that unpacks the
    source (into SRC), applies the appropriate patches,
    creates a build dir under BUILD, and creates an inst
    dir under INST.  Configures, builds, installs (into
    INST/<pkg>/usr/* -- this includes copying
    ORIG/zlib*postinstall into
    INST/<pkg>/etc/postinstall/zlib.sh),
    and creates packages.  Note that THIS file doesn't go
    into a "SCRIPTS/zlib" subdir -- while anything installed
    or created in the other CAPITAL directories does go into
    a zlib subdir.   The reason?  There will never be more
    than ONE build script for any given package.

And that's (pretty much) it.

Why do I want zlib-build to handle so much?  Why do I want so much stuff in 
ORIG, and NOT created within the unpacked src dir by .patch?

It's very difficult to add a file to a package, AND munge the Makefile's to 
install it.  Okay, it's not *technically* difficult -- for non-autotool 
packages you just *do* it.  For autotool packages, you have to add the new 
file to the Makefile.am, then re-run automake and autoconf -- which leads 
to a LOT of changes in a LOT of the files == BIG patch.

But in both cases, these changes are NOT ultimately destined for upstream 
inclusion.  They are cygwin-only, and will never make it into the "real" 
package.  IMO, .patch should be limited to code changes that are 
upstream-compatible.  Or at least to those changes that aren't frufru.

Heck, in most of cgf's packages, the README (if it exists) is just 
hand-copied into his inst dir immediately before tar'ring -- and is often 
forgotten (binutils? <g>)  Adding a file to a package The Right Way(tm) is 
just plain HARD.  Why enforce extra work?  (BTW, I now officially hate the 
"put patch file into CYGWIN-PATCHES" idea -- even though it was mine, 
originally.  I'm always forgetting to remove it before creating the new 
patch, or forgetting to put it back in before creating the src tarball.  Ugh.)

Using a build script (zlib-build) allows us to have multiple patches, 
applied in the proper order.  We don't need to "revert" to pristine source, 
'cause that's what we're distributing.  AND, this sort of structure is 
easily translated to dpkg or rpm style tools, if we ever go there.  (Plus, 
scripts -- once they are properly written -- don't make mistakes, or forget 
to include the patch in the src tarball, etc)

My proposal also minimizes the hurdle that new porters must overcome -- IF 
we (I?) provide a "sample" build script.  Any other plan requires porters 
to do a LOT of work.  For instance, I don't know how **I** would get 
readline or ncurses to build with a single patch and one "configure; make; 
make install" run.  Currently, I do *two* make; make install runs -- and in 
the readline case, there's a secondary patch applied before the second 
(DLL-building) make; make install.  Worse, I don't see this changing until 
libtool supports the new dll-building/auto-import stuff in binutils (or 
whatever gcc profile-based alternative Robert comes up with)

The point?  Simpler src packaging standards don't allow for these complex 
builds -- and in my experience, ports of new packages seem to always go 
thru a stage where building is complicated.  Then later, as the port is 
refined (*) the build becomes simpler.

(*)more stuff accepted by upstream maintainers, better autotool integration 
-- not to mention ld improvements (past) and libtool improvements (future).

Do we want a simple src packaging standard -- that serves as a disincentive 
for new package ports, because it doesn't allow for complex builds?  "No, 
I'm sorry -- you can't submit that package until you can get it to build 
using this src packaging standard." "But I can't -- I need to wait for this 
change to autoconf and that change to binutils -- THEN I can get it to 
build 'simply'"  "Sorry, you must table your package.  Go work on autoconf 
and binutils, then come back"

too much delayed gratification.  If I had had to WAIT until libtool was 
working nicely so that "./configure --build-shared --build-static" worked 
for ncurses on cygwin, we'd still be waiting.

Closer to home: if we go with an extrememly simple src packaging standard, 
I won't be able to update readline for a LONG time (unless I get a 
"waiver") -- because I won't be able to support the current ugly build 
procedure of readline under a new -src standard if it isn't 
flexible/powerful enough.  (Oh, yeah -- and then there's the multiple 
packages from a single source problem as in ncurses/libncurses:  Can't be 
done without a well-written build script.  In the simpler src packaging 
standards, where would IT go?  Right now it's in CYGWIN-PATCHES, but only 
for lack of a better place)

The danger is in making the "complex" -src standard too difficult.  Well, 
since my proposal is "put these files in those places, and use a build 
script to do all the work" and "oh BTW, here's a sample build script, 
adjust to suit your project" I think it isn't too difficult.

And it is quite similar to the structure used by rpm (dpkg) so it should 
facilitate moving to that at some point in the future, if we decide to do so.

Then the answer to "How do I build cgywin from CVS" becomes

cd /usr/src/cygwin/SRC
cvs co winsup
mv src cygwin-cvs
cp cygwin-cvs/winsup/maint/cygwin-cvs-build /usr/src/cygwin/SCRIPTS/
cd /usr/src/cygwin/SCRIPTS
./cygwin-cvs-build doitall

and poof! /usr/src/cygwin/PACKAGES/cgywin-<date>.tar.bz2 and 
cygwin-<date>-src.tar.bz2.  And cygwin-<data>-src.tar.bz2, naturally, 
follows the -src standard, and contains its own cygwin-build script 
(slightly different from the cgywin-cvs-build script, so this requires two 
similar build scripts in winsup/maint.  Think about it, you'll realize why)

--Chuck


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]