This is the mail archive of the cygwin-apps@cygwin.com mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

Re: patches to vendor source trees - discussion


On Sun, 2001-11-04 at 16:39, Charles Wilson wrote:
> Robert Collins wrote

> > 1) setup.exe isn't ready for more than bare bones changes - regardless
> > of rpm or dpkg inspiration.
> 
> 
> I didn't want setup to do anything but unpack "foo-ver-rel-src.tar.bz2" at 
> the "/usr/src" level.  foo-ver-rel-src.tar.bz2 contains the ORIG/foo/* and 
> DOC/foo/* SCRIPTS/* structure.  I realize now that I specified full 
> '/'rooted paths in my original email; but I didn't really intend that the 
> tarballs include the "/usr/src" part.

k. That makes some stuff more clear :}.
 
> *PERHAPS* -- if it doesn't already -- setup could record the fact in 
> /etc/setup/foo-src.list.gz that "foo-pristine-ver.tar.gz" was put into 
> /home/rcollins/src/ORIG/foo/ (or /usr/src/ORIG/foo).  Likewise the other 
> primary files in the foo-ver-rel-src.tar.bz2 archive.
>   But this really isn't a big deal.

As I mentioned, I don't think it should record source stuff at all -
with the specific exception of packages whose binary package *is* a
source package (a specific example of which is the kernel-source
packages on linux systems).
 
> 
> version   0......................1..2.34
> 
> The baby step scheme is
> 
> version   0..1..2..3..4..5..6..7..8..9..10
> 
> I like asymptotic approaches, not linear ones.  (Also, we get "there" with 
> fewer iterations, usually...of course this depends on how "baby" your baby 
> steps are)
 
By baby, I mean small enough to mitigate the risk of totally stuffing
things up. As big as possible in all other respects.
 
> > I'd love too be able to do that - with with the rpm or deb inspiration,
> > but frankly setup ain't ready for that.
> Ready for what?  All I'm saying is "maintainers, make your src package look 
> like this".  All setup has to do is unpack it.  Under /usr/src (or ~/src if 
> you like)
 
rpm/deb have a bunch of support infrastructure for package maintainers -
if setup could install rpms or debs then we could use those tools.
 
> Dunno much about .deb; I tend to lump all the automated build systems together.

I've played with the BSD ports tree, the debian package system, and the rpm package system. They are similar, but they do have differences.

> Where does stuff ACTUALLY get built?  in /cygwin1.3/* ?  Ick!  I'd imagine 
> your '/' directory (wherever it happens to be) gets awful cluttered with 
> oodles of foo2.4* bar4.6* and baz1.3* files and directories...

Most things build in the source tree, but the package build can be done
in a .build subdir or equivalent trivially.


> > Nope. IMO it shouldn't. The cygbuild script or whatever we call it, that
> > can call setup to grab the source, 
> Okay, you're assuming setup has the argv handling stuff already...

Well, it does - it's got getopt now, just no options :]. More seriously,
it doesn't have the flexability to do this regardless of cmd line
parameters yet (because of the hard coded /usr/src)
 
 
> Yes!  But I wasn't going that far.  Cygbuild is in the future (say, version 
> 3 in my scheme above).  Version 1 -- packages have their own specific build 
> script.  Later, that build script is converted into a .spec-like file or 
> rules-like file for cygbuild to parse.  Or something.

FYI debian 'rules' is a make file. The first line reads
#! /bin/make -f
with the obvious consequence.

> Anyway, "my" proposed -src package only contains:
> 
> /cygwin/ORIG/zlib/zlib-113.tgz
> /cygwin/ORIG/zlib/zlib-1.1.3-9.patch
> /cygwin/ORIG/zlib/zlib-1.1.3-9.postinstall
> /cygwin/ORIG/zlib/*
> /cygwin/DOCS/zlib/zlib-1.1.3.README
> 
> /cygwin/SCRIPTS/zlib-build
 
Right, what I object to here is the pathing:
cygwin/orig|docs+|scripts/zlib

to me that means I cannot separate out building separate packages unless
I descend into the cygwin/ORIG directory. Also the zlib dir for ORIG and
DOCS is redundant, as all the files are package-version and
package-version-foo in naming.


> setup-uninstall should just remove those files.  Not the BUILD/zlib or INST/zlib or other "created" files

Again, for source packages, IMO setup should not do anything uninstall wise. Thats the users call.  
 
> > My point on this was that the patch for the end user, and how the
> > maintainer manages their patches are orthogonal. A one liner to create a
> > new patch at package time can be scripted, and won't add to your
> > management overhead.
> 
> 
> Not really.  If you make it hard to map the maintainer's method to the 
> "official" packaging standard, then you are creating an artificial barrier 
> to new ports/maintainers.  Maximum flexibility says: "put your patches here 
> (where they won't get lost) and make sure your build script will apply them 
> properly".

I don't see how I'm making it hard in my suggestion... I'm suggesting
that the source tarball be *the vendors tarball* (given we offer
pristine source).

And that we have a *single* additional file created via.
diff -Nrup origsrcdir package-vendor > package-vendor-suffix.patch
 
In my original suggestion I was suggesting that the patch be added to
the vendor tarball, but that suggestion also included pre-patched
source. If the source isn't pre-patched having the patch separate is
easier for the maintainer, I think.

> Your idea makes it difficult for *me* to keep track of all the various 
> patch files -- or to explain to a new maintainer.  I'd have to maintain a 
> *separate* build script that handled the *separated* patches, but creates a 
> src tarball with a unified patch and a build script that uses it. (I'm 
> thinking along the lines of "how do I ensure that someone else can take 
> over maintaining my 'readline' or 'libtiff' package if I get hit by a bus". 
>   With the "all-patches-in-one" method, it would be practically impossible 
> for a NEW maintainer to, say, remove the lossless jpeg stuff from the jpeg 
> package, but keep the dllization patches.)

*sigh*. But I'm not suggesting that you elimintate your separate
patches. 

I'm suggesting that the user have a consolidated patch for them to use,
and that your working patches can be contained therin should you need
the, - preapplied or not at your choice.

> I'd like for anybody to be able to take over maintainance of ANY package if 
> the original maintainer goes kaflooey, using ONLY the stuff in the -src 
> package.

Thats a worthy goal. I'd hate to make it *hard* to do this.
 

> > -ONLY if your autotools version is different- (or for old autoconfs :[).
> 
> Well, that's pretty likely.  Notwithstanding my autotool harmonization 
> packages, there are packages out there that use autoconf 2.13, 2.50, and 
> 2.52 (and soon 2.6 and 3.0).  There probably aren't many packages that 
> haven't been updated to 2.13 -- the four year 2.50 development cycle was 
> long enough to flush 2.12 out of the pipeline. <g>
> 
> automake 1.3 still sees use, and there are six (count 'em, six) subversions 
> of 1.4 that have different Makefile.in signatures: 1.4, 1.4p1,... 1.4p5. 
> And don't forget 1.5.

Yes but!. Automakte 1.4p5 should run automake 1.3 stuff just fine, and
1.5 is the fly in the ointment.
 
> > for autotools stuff, the src has to depend on having the correct
> > autotools installed.
> Which one is "correct"?  That'll be different for each package (worst case)

Using debian as an example, they get by with two versions and the wrapper script.
 
> > then the patch only alters Makefile.am and configure.in. 'nuff said.
>
> You are assuming that people who want to build from (our) -src packages 
> MUST run autoreconf and/or automake themselves, before configuring.  That's 
> contrary to the autotools philosophy: distributed packages should be 
> buildable even if the builder does NOT have autotools installed.  (That's 
> why libtoolize puts the libtool scripts into the target package itself, 
> instead of assuming that libtool itself will be installed on the builder's 
> machine)

No, I'm assuming that for *your* packages, you wanted to trade patch
size for complexity. Personally, I'd patch the whole kit and kaboodle,
and trust gz to make it managable size wise.
 
> > yees, I see what you mean. However you have an environment where
> > building the package as it gets installed requires a fixed diretory tree
> > - information not in the src dir.
> 
> Okay, but so does debian, right?  

No, debian package building is self contained. There is a new directory
called debian sitting under the top src dir. In that there is (letsee,
automake 1.5 is a good example,
changelog
control
copyright
doc-base
lintian.overrides
postinst
prerm
rules

But thats debian, and I intended to simply talk about what setup
extracts - what the user expects to see, and we are now talking about
how the packages are created by the user.

> > You're also putting the src dir at a fixed level in the tree relative to
> > the package data - which might play havoc with nested source trees like
> > winsup+newlib. 
> Err...only if those are ever downloaded and built independently of each 
> other.  I'm assuming they will not be.

Could be risky assumption :]. Setup.exe will build quite happily out of
the tree AFAIK. w32api certainly will....
 
> > If we make the extra work easy, then is it HARD? Seriously, I'll give
> > you a ssh account here, and you can have a play with the debian package
> > building environment. Nothing is easier - for the maintainer - IMO.
> 
> 
> Well, it's already pretty easy -- for an autotool'ed project -- if you 
> understand .am files and .m4 files.  But ordinary porters should NOT have 
> to learn the intricacies of m4 scripting just to insure that package foo 
> installs its cygwin-specific README file.  Hell, they shouldn't have to 

Cygwin packagers have to learn (this is from experience - watching you,
doing squid..)
* text vs binmode coding. (Can anyone show automode doing the job
completely, ever?)
* .dll building for packages with libraries. (libtool isn't complete
yet).
* Makefile twitching. (make foo.exe fails - oh, it only has a rule for
make foo).

And thats a minimum. Frankly, adding a automake target for installing a
README is trivial.

> this is madness (to borrow a phrase).
 
Perhaps :}. I'm really not intending to push the barrier up, but I do
think that README.Cygwin is really just the first little bit to be a
maintainer.
 
> > So how many build scripts do we need? And how do they determine which
> > patches to apply?
> 
> At *first* (step 1, but not a baby step <g>), one build script for each 
> package.  So, zlib-build.  libpng-build.  automake-build.  etc. (versioning 
> aside -- see below)  Later, those get converted into, what?, zlib.cyg, 
> libpng.cyg, and automake.cyg.  cygbuild parses them, etc. Just like 
> dpkg/rules or rpm/spec.

See my comment on 'rules' before, but yes. I have no issue with this. (I
*still* maintain it's orthogonal to having a global patch or not though
:}) 
 
> > Ok. I thought mine did as well... guess not. 
> 
> 
> Well, I dunno.  Everybody has their own build style.  Mine pretty much 
> sucks (no training).  I bet your proposal works great for your style, but 
> sucks for mine -- and my proposal works great for my style, but sucks for 
> yours.  :-)
 
Granted.
 
> > Ok here's what I think we are agreeing on:
> > 1) The package should be able to be recreated by anyone downloading the
> > source
> > 2) We should distribute pristine source.
> > 3) We need to allow an arbitrarily complex build process decided by the
> > maintainer
> 
> 
> Yep.
> 
> 
> > Your proposal (key points):
> > * in /usr/src/Cygwin create a few dirs and place the relevant items for
> > each tool in the appropriate dir.
> 
> 
> Okay, but let's just say "cygwin/" is stored in the tarball.  Assume 
> setup.exe unpacks it in /usr/src -- but the tarball *should* be relocatable.
> > * provide a sample script for new packages to learn from
> > * Provide the patches needed to *allow the script* to convert the source
> > from pristine to cygwin-ready.
> 
> 
> Yep.
> 
> 
> > My proposal (updated) (key points):
> > * include in the directory above the extracted source, a consolidated
> > patch that alters the pristine source to be ready for building into a
> > package. 
> > * The patch should add a README.Cygwin to the $(top_srcdir)/cygwin
> > directory at a minimum.
> > 
> > Now, I believe that they *both* meet the things we agree on above. To
> > address your questions about my proposal,
> > * how are the requirements for building different packages from one
> > source served?
> > Thats up to the maintainer. If we? (you?) provide a couple of example
> > scripts new maintainer will have a good starting point. The mechanism
> > could be as easy as make (build-src| build-libs| build-binary )
> 
>                        ^^^^
> UUGGHH! You want to muck with the Makefile.am, Makefile.in, Makefile's to 
> support building *our* split-up packages directly via make?  I *really* 

No! I don't want that at all. I want to let *you* choose, just as you do
now. I didn't want to touch the package creation stuff *at all* other
than specify that a global patch be provided.

As for building via make, I think that when we start to address
automated package creation, thats one of the things to address. I happen
to like the debian way, but please note: It doesn't require many
autotool file fiddling, because it uses a separate makefile, that calls
the pacakge makefile, and then does stuff on top of it - just like a
script, but more flexible for the user.


> tar cvjf ../mingw-new.tar.bz2 usr/bin/mingwm10.dll \
>    usr/include/mingw usr/lib/mingw
> tar cvjf ../w32api-new.tar.bz2 usr/include/w32api \
>    usr/lib/w32api
> tar cvjf ../cgywin-new.tar.bz2 usr --exclude='usr/bin/mingwm10.dll' \
>    --exclude='usr/include/mingw' --exclude='usr/lib/mingw' \
>    --exclude='usr/include/w32api' --exclude='usr/lib/w32api' \
>    --exclude='usr/bin/runtest'
> 
> I really want to avoid imposing "learn autotools and .am and .m4" on all 
> porters.

Sure. So to do this as a makefile:
mingw_src:
  tar cvjf ../mingw-new.tar.bz2 usr/bin/mingwm10.dll \
    usr/include/mingw usr/lib/mingw

w32api:
...

but *not* in the packages makefile.
 
> > * What about complex packages?
> > Well, thats the beauty, by including the scripts within the source tree
> > they can't get lost. So it becomes orthogonal to the 'standard'.
> 
> 
> Ah, but I'm assuming that *THIS* is the source tree:
> 
> cygwin/ORIG/foo.tgz
> cygwin/ORIG/foo.patch1
> cygwin/ORIG/foo.patch2
> cygwin/SCRIPTS/foo-build
> cygwin/DOCS/foo.README
> 
> You're assuming that the "unpacked" foo/ directory is the source tree.  But 
> in either case, the patches won't get lost.

But! the user cannot grab the foo tree and mv it elsewhere in one step.
Not without grabbing *all* the source trees. URGGH.
 
> > What I don't like about your idea is:
> > 1) fixed directory location. I like to do this stuff in ~/src, not
> > /usr/src. Others may have their own preferences.
> Misunderstanding.

K.
 
> > 2) We'll end up with ??how many ?? different scripts in that directory
> > tree? (remember, as each -suffix will have a different script (think on
> > this if you need to) installing the source to three revisions of package
> > foo ends up with three foo-build's, three foo-README's etc.
> Erm...and what's wrong with that?  debian has how many dsc files for 
> different versions of the same package?  I don't see the problem.

One for each, and because they are in the same directory as the src tree
is rooted at, there is no need to fossack around finding all the related
files.

> As I see it, the debian method eliminates my:
> 
> cgywin/PACKAGES/
> cgywin/ORIG/
> cgywin/SRC/
> 
> directories and just uses the top level (call it cgywin/) for the files 
> that would go into them.  It also collapses the cygwin/SRC up to the 
> cygwin/ level, and unpacks the tarball there.
> 
> the debian scheme doesn't seem to have any provision for building *outside* 
> the unpacked source directory (unless it uses some dynmically created dir 
> in /var/tmp ?).  ditto the temporary installation directory.  So that's
>

> cygwin/BUILD
> cygwin/INST
> 
> finally, the
> 
> cygwin/SCRIPTS/
> cygwin/DOCS/
> 
> are kinda replaced by the cygwin/<pkgdir>/debian/ directory (created by the 
> patch) coupled with the .dsc file at the cgywin/ level.

whats the "cygwin" dir? 
 
> do I have that right?  If so, then the primary difference between the 
> proposals is
> 
> a) splitting things up into multiple toplevel directories vs. everything 
> goes in one toplevel directory (not counting the unpacked source archive)

Uhmm possibly, if I uderstand you right. You seem to be using 1 top
level dir, and ~3 1st level dirs. I'm suggsting 1 top level dir per
package.
 
> b) script handles applying (possibly) multiple patches; some support files 
> NOT created by a patch but are included directly in the primary -src 
> archive  vs.  single patch, which creates SOME support files within the 
> patchedNunpacked src tree, one other support file (.dsc) is NOT created by 
> that unified patch.


 
> I gotta say, this (b) is one area where I think RPM-style wins out over 
> DEB...ever look at a linux kernel-src RPM?  *Much* more informative than if 
> everything were all in a single patch.  IMO.  <g>

kernel source is a different beast. Ever look at debian  for this? All
the kernel patches *are* separate. They come as separate .debs and are
automatically applied by make-kpkg.
 
> However, in both cases the ultimate goal is to transition to a "process 
> description file" that is interpreted by a custom tool (cygbuild? debbuild? 
> etc)

Uhmm yes.


Thining about this:

Here are my revised immediate goals (forget everything I wrote above):
1) ship a single -src tarball that contains everything needed to build a
package.
2) Ship unaltered vendor src in that tarball for maintenance ease.
3) Not impose any new rules on the package maintainers.

To this end I proposed that we include a single unified patch, and the
src tree.

The reason I don't want to alter the rules for package maintainers is
that  I agree with the spirit of all you've said, and I percieve the
creation of these scripts, and the directory structure, and teh tools to
be a big task. We have a working status quo. The various patches go into
a CYGWIN_PATCHES or wahtever dir in the source tree, and the README
tells how to create teh package. That works. Lets *leave it in place*.

Why? because any change now that is not a direct step towards deb/rpm is
simply bit twiddling, IMO. It won't make your life easier, and it will
raise the bar for maintainers.

When we are able to install an rpm or a deb, *then* I believe we should
start the change process. And then the steps will be pretty clear.

So there it is, I wasn't actually trying to change anything signficiant,
other than making the ability to go (to originally, now from ) pristine
source to cygwin ready source a standard step. The rest of the process
is just fine until the infrastructure to do more is in place iMO.

Rob






Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]