Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance

[Yoni Londner <yonihola2 at gmail dot com>]

Hi,

Following is a cygwin stat performance testing appilication.

It compares Cygwin's stat() with Native NT.

To compile it: gcc -g cygtest.c -o cygtest.exe -lntdll -lwinmm

It tests:

- lstat(unpatched): Cygwin's lstat implementation, without my 
performance patch.

- lstat(patched): Cygwin's lstat implementation, with my performance patch.

- GFA: GetFileAttributes(): This NT API allows getting information on a 
file by name without having to open a handle to the file. This misses 
information such as st_ino and st_nlink, but is added here for sake of 
comparison. Theoretically this should have been the fastest API, but on 
Win7 it is actually slower that 
CreateFile(dir)+QueryDirectoryFile+CloseFile. GFA takes 0.01ms on XP and 
0.16 on Win7 (strange...).

- QDF: QueryDirectoryFile(): This NT API allows getting all the 
information except for st_nlink (it does give st_ino and all other 
information). QDF is 0.03ms on XP and 0.046ms on Win7.

- QIF: QueryInformationFile(): This NT API allows getting st_ino and 
st_nlink, but requires opening a handle opening to the file itself. This 
is what Cygwin uses in its code to implement lstat(), and as you see, 
its the slowest method. QIF is 0.92ms on XP and 0.17ms on Win7.

And here are the results of my tests on /bin ~3500 files:
   XP: CYGWIN_NT-5.1 yoni 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
   lstat(1.7.5 unpatched) 3587 files stat() 490.2ms, per file: 0.1367ms
   lstat(1.7.5 patched) 3585 files stat() 78.12ms, per file: 0.02179ms
   lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
   lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
   GFA 3585 files stat() 38.09ms, per file: 0.01062ms
   QDF 3585 files stat() 105.5ms, per file: 0.02942ms
   QIF 3585 files stat() 3309ms, per file: 0.9229ms

   Win7: CYGWIN_NT-6.1 bush-PC 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 
Cygwin
   lstat(1.7.5 unpatched) 3457 files stat() 934.1ms, per file: 0.2702ms
   lstat(1.7.5 patched) 3455 files stat()  634ms, per file: 0.1835ms
   lstat(1.7.7 unpatched) 3459 files stat()  777ms, per file: 0.2246ms
   lstat(1.7.7 patched) 3459 files stat()  631ms, per file: 0.1824ms
   GFA 3455 files stat()  574ms, per file: 0.1661ms
   QDF 3455 files stat()  159ms, per file: 0.04602ms
   QIF 3455 files stat()  599ms, per file: 0.1734ms

The summary of all this above info:
- GFA() is fastest only on XP - but since it gives out only partial 
info, it should only be used on XP for 'cache validation' (if we start 
caching in Cygwin file information from scandir->lstat).

- QDF() MUCH MUCH MUCH faster than QIF(35x faster on XP, 4x faster on Win7).

The only information we 'loose' by using QDF() vs. QIF() is st_nlink.

Giving an option to 'give up' on st_nlink and in return giving 35x (XP) 
or 4x (Win7) performance improvement is very worth while!

In our company, after we moved to use the patched cygwin1.dll with the 
improved performance, no developer wants to go back, since everything 
works just SO much faster: opening a shell, grep, compilation (make), 
bash scripts...

Regaring compatiblity for loadable filesystems on Windows (MVFS etc...): 
Since cygwin knows the volume information, it can use QDF() for NTFS and 
FAT volumes ('white list'). Since these are implemented in NTFS.SYS and 
FAT.SYS - which we know are implemented correctly, there is no problem, 
and this covers 99% of the FS system calls. The others can be added once 
we see they work correctly.

Yoni

Code:
============================================
#include <stdlib.h>
#include <stdio.h>
#include <dirent.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#include <windows.h>
#include <mmsystem.h>
#include <ntdef.h>
#include <wchar.h>

/* to compile: gcc -g cygtest.c -o cygtest.exe -lntdll -lwinmm */
/* results on /bin ~3500 files:
   XP: CYGWIN_NT-5.1 yoni 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
   lstat(1.7.5 unpatched) 3587 files stat() 490.2ms, per file: 0.1367ms
   lstat(1.7.5 patched) 3585 files stat() 78.12ms, per file: 0.02179ms
   lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
   lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
   GFA 3585 files stat() 38.09ms, per file: 0.01062ms
   QIF 3585 files stat() 3309ms, per file: 0.9229ms
   QDF 3585 files stat() 105.5ms, per file: 0.02942ms

   Win7: CYGWIN_NT-6.1 bush-PC 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 
Cygwin
   lstat(1.7.5 unpatched) 3457 files stat() 934.1ms, per file: 0.2702ms
   lstat(1.7.5 patched) 3455 files stat()  634ms, per file: 0.1835ms
   lstat(1.7.7 unpatched) 3459 files stat()  777ms, per file: 0.2246ms
   lstat(1.7.7 patched) 3459 files stat()  631ms, per file: 0.1824ms
   GFA 3455 files stat()  574ms, per file: 0.1661ms
   QIF 3455 files stat()  599ms, per file: 0.1734ms
   QDF 3455 files stat()  159ms, per file: 0.04602ms
 */

typedef long long i64;
typedef unsigned long long u64;
typedef unsigned char u8;
typedef unsigned int u32;

#define do_err(...) \
do { \
    printf("%s:%d: ", __FUNCTION__, __LINE__); \
    printf(__VA_ARGS__); \
    exit(1); \
} while (0)

typedef struct _FILE_NETWORK_OPEN_INFORMATION {
  LARGE_INTEGER  CreationTime;
  LARGE_INTEGER  LastAccessTime;
  LARGE_INTEGER  LastWriteTime;
  LARGE_INTEGER  ChangeTime;
  LARGE_INTEGER  AllocationSize;
  LARGE_INTEGER  EndOfFile;
  ULONG  FileAttributes;
  ULONG  Unknown;
} FILE_NETWORK_OPEN_INFORMATION, *PFILE_NETWORK_OPEN_INFORMATION;

typedef struct _FILE_BASIC_INFORMATION {
  LARGE_INTEGER           CreationTime;
  LARGE_INTEGER           LastAccessTime;
  LARGE_INTEGER           LastWriteTime;
  LARGE_INTEGER           ChangeTime;
  ULONG                   FileAttributes;
} FILE_BASIC_INFORMATION, *PFILE_BASIC_INFORMATION;

typedef struct _FILE_STANDARD_INFORMATION {
    LARGE_INTEGER AllocationSize;
    LARGE_INTEGER EndOfFile;
    ULONG NumberOfLinks;
    BOOLEAN DeletePending;
    BOOLEAN Directory;
} FILE_STANDARD_INFORMATION, *PFILE_STANDARD_INFORMATION;

typedef struct _FILE_POSITION_INFORMATION {
    LARGE_INTEGER CurrentByteOffset;
} FILE_POSITION_INFORMATION, *PFILE_POSITION_INFORMATION;

typedef struct _FILE_INTERNAL_INFORMATION {
    LARGE_INTEGER IndexNumber;
} FILE_INTERNAL_INFORMATION, *PFILE_INTERNAL_INFORMATION;

typedef struct _FILE_EA_INFORMATION {
    ULONG EaSize;
} FILE_EA_INFORMATION, *PFILE_EA_INFORMATION;

typedef struct _FILE_ACCESS_INFORMATION {
    ACCESS_MASK AccessFlags;
} FILE_ACCESS_INFORMATION, *PFILE_ACCESS_INFORMATION;

typedef struct _FILE_MODE_INFORMATION {
    ULONG Mode;
} FILE_MODE_INFORMATION, *PFILE_MODE_INFORMATION;

typedef struct _FILE_ALIGNMENT_INFORMATION {
    ULONG AlignmentRequirement;
} FILE_ALIGNMENT_INFORMATION, *PFILE_ALIGNMENT_INFORMATION;

typedef struct _FILE_NAME_INFORMATION {
    ULONG FileNameLength;
    WCHAR FileName[1];
} FILE_NAME_INFORMATION, *PFILE_NAME_INFORMATION;

typedef struct _FILE_ALL_INFORMATION {
  FILE_BASIC_INFORMATION     BasicInformation;
  FILE_STANDARD_INFORMATION  StandardInformation;
  FILE_INTERNAL_INFORMATION  InternalInformation;
  FILE_EA_INFORMATION        EaInformation;
  FILE_ACCESS_INFORMATION    AccessInformation;
  FILE_POSITION_INFORMATION  PositionInformation;
  FILE_MODE_INFORMATION      ModeInformation;
  FILE_ALIGNMENT_INFORMATION AlignmentInformation;
  FILE_NAME_INFORMATION      NameInformation;
} FILE_ALL_INFORMATION, *PFILE_ALL_INFORMATION;

typedef struct _FILE_ID_BOTH_DIRECTORY_INFORMATION {
    ULONG         NextEntryOffset;
    ULONG	  FileIndex;
    LARGE_INTEGER CreationTime;
    LARGE_INTEGER LastAccessTime;
    LARGE_INTEGER LastWriteTime;
    LARGE_INTEGER ChangeTime;
    LARGE_INTEGER EndOfFile;
    LARGE_INTEGER AllocationSize;
    ULONG         FileAttributes;
    ULONG         FileNameLength;
    ULONG         EaSize;
    CHAR          ShortNameLength;
    WCHAR         ShortName[12];
    LARGE_INTEGER FileId;
    WCHAR         FileName[0];
} FILE_ID_BOTH_DIRECTORY_INFORMATION, *PFILE_ID_BOTH_DIRECTORY_INFORMATION;

typedef struct _IO_STATUS_BLOCK {
    union {
        NTSTATUS Status;
        PVOID Pointer;
    };
    ULONG_PTR Information;
} IO_STATUS_BLOCK, *PIO_STATUS_BLOCK;

typedef enum _FILE_INFORMATION_CLASS {
  FileDirectoryInformation = 1,
  FileFullDirectoryInformation,
  FileBothDirectoryInformation,
  FileBasicInformation,
  FileStandardInformation,
  FileInternalInformation,
  FileEaInformation,
  FileAccessInformation,
  FileNameInformation,
  FileRenameInformation,
  FileLinkInformation,
  FileNamesInformation,
  FileDispositionInformation,
  FilePositionInformation,
  FileFullEaInformation,
  FileModeInformation,
  FileAlignmentInformation,
  FileAllInformation,
  FileAllocationInformation,
  FileEndOfFileInformation,
  FileAlternateNameInformation,
  FileStreamInformation,
  FilePipeInformation,
  FilePipeLocalInformation,
  FilePipeRemoteInformation,
  FileMailslotQueryInformation,
  FileMailslotSetInformation,
  FileCompressionInformation,
  FileObjectIdInformation,
  FileCompletionInformation,
  FileMoveClusterInformation,
  FileQuotaInformation,
  FileReparsePointInformation,
  FileNetworkOpenInformation,
  FileAttributeTagInformation,
  FileTrackingInformation,
  FileIdBothDirectoryInformation,
  FileIdFullDirectoryInformation,
  FileValidDataLengthInformation,
  FileShortNameInformation,
  FileMaximumInformation
} FILE_INFORMATION_CLASS, *PFILE_INFORMATION_CLASS;

#define NSEC_PER_MS 1000000
#define SHARED_DATA ((KSHARED_USER_DATA *)0x7ffe0000)
#define STATUS_NO_MORE_FILES ((NTSTATUS)0x80000006L)

typedef struct _KUSER_SHARED_DATA {
    unsigned long TickCountLowDeprecated;
    unsigned long TickCountMultiplier;
    volatile unsigned long long InterruptTimeQuad;
} KSHARED_USER_DATA;

static u64 time_monotonic_nsec(void)
{
    u64 t;
    while ((t = SHARED_DATA->InterruptTimeQuad)!=
	SHARED_DATA->InterruptTimeQuad);
    return t*100;
}

static void usage(void)
{
    fprintf(stderr, "stat perf: cygtest METHOD [dir]\n"
	"  lstat: Cygwin's lstat\n"
	"  GFA: GetFileAttributes\n"
	"  QIF: QueryInformationFile\n"
	"  QDF: QueryDirectoryFile\n"
	"\n"
	"bash: uname -a; for i in lstat GFA QIF QDF ; do ./cygtest $i ; "
	"done\n");
    exit(1);
}

NTSTATUS NTAPI NtQueryFullAttributesFile(POBJECT_ATTRIBUTES 
ObjectAttributes,
    PFILE_NETWORK_OPEN_INFORMATION FileInformation);
NTSTATUS NTAPI RtlAnsiStringToUnicodeString(
    PUNICODE_STRING DestinationString, PANSI_STRING SourceString,
    BOOLEAN AllocateDestinationString);
NTSTATUS NTAPI NtQueryAttributesFile(
    POBJECT_ATTRIBUTES   ObjectAttributes,
    PFILE_BASIC_INFORMATION FileAttributes);
NTSTATUS NTAPI NtQueryDirectoryFile(HANDLE FileHandle, HANDLE Event,
    void *ApcRoutine, PVOID ApcContext, PIO_STATUS_BLOCK IoStatusBlock,
    PVOID FileInformation, ULONG Length,
    FILE_INFORMATION_CLASS FileInformationClass, BOOLEAN ReturnSingleEntry,
    PUNICODE_STRING FileMask, BOOLEAN RestartScan);
NTSTATUS NTAPI NtOpenFile(PHANDLE FileHandle, ACCESS_MASK DesiredAccess,
  POBJECT_ATTRIBUTES ObjectAttributes, PIO_STATUS_BLOCK IoStatusBlock,
  ULONG ShareAccess, ULONG OpenOptions);
NTSTATUS WINAPI NtClose(HANDLE Handle);
NTSTATUS NTAPI NtCreateFile(PHANDLE FileHandle, ACCESS_MASK DesiredAccess,
  POBJECT_ATTRIBUTES ObjectAttributes, PIO_STATUS_BLOCK IoStatusBlock,
  PLARGE_INTEGER AllocationSize, ULONG FileAttributes, ULONG ShareAccess,
  ULONG CreateDisposition, ULONG CreateOptions, PVOID EaBuffer OPTIONAL,
  ULONG EaLength);
NTSTATUS NTAPI NtQueryInformationFile(HANDLE FileHandle,
    PIO_STATUS_BLOCK IoStatusBlock, PVOID FileInformation,
    ULONG Length, FILE_INFORMATION_CLASS FileInformationClass);


WCHAR *wstr_from_str(WCHAR *wstr, char *utf)
{
    WCHAR *wc = wstr;
    u8 *u = (u8 *)utf;
    u32 t;
    int i = 0;
    while (*u)
    {
	u8 c = u[0];
	if (c<0Xc0)
	{
	    wc[i++] = c;
	    u++;
	}
	else if (c<0xe0)
	{
	    if ((u[1] & 0xc0)!=0x80)
	        goto Invalid;
	    t = ((c & 0x1f) << 6) | (u[1] & 0x3f);
	    wc[i++] = (WCHAR)t;
	    u += 2;
	}
	else if (c<0xf0)
	{
	    if ((u[1] & 0xc0)!=0x80 || (u[2] & 0xc0)!=0x80)
	        goto Invalid;
	    t = ((c & 0x0f) << 12) | ((u[1] & 0x3f) << 6) | (u[2] & 0x3f);
	    wc[i++] = (WCHAR)t;
	    u += 3;
	}
	else
	    goto Invalid;
	continue;
Invalid:
	wc[i++] = '?'; /* invalid */
	u++;
    }
    wc[i] = 0;
    return wstr;
}

static HANDLE get_directory_handle(char *dir)
{
    wchar_t *wdir;
    IO_STATUS_BLOCK io;
    UNICODE_STRING udir;
    OBJECT_ATTRIBUTES attr;
    HANDLE hdir;
    char win_dir[512], *p;
    int ret;
    sprintf(win_dir, "\\??\\c:/cygwin%s", dir);
    for (p = win_dir; *p; p++)
    {
	if (*p=='/')
	    *p = '\\';
    }
    wdir = malloc(sizeof(WCHAR)*(strlen(win_dir)+1));
    wstr_from_str(wdir, win_dir);
    udir.Buffer = wdir;
    udir.Length = wcslen(udir.Buffer) * sizeof(WCHAR);
    udir.MaximumLength = udir.Length + sizeof(WCHAR);
    InitializeObjectAttributes(&attr, &udir, 0, NULL, NULL);
    if ((ret = NtOpenFile(&hdir, SYNCHRONIZE|FILE_LIST_DIRECTORY,
	&attr, &io, FILE_SHARE_VALID_FLAGS,
	FILE_SYNCHRONOUS_IO_NONALERT|FILE_OPEN_FOR_BACKUP_INTENT
	|FILE_DIRECTORY_FILE)))
    {
	do_err("NtOpenFile failed(%s): %d\n", win_dir, ret);
    }
    return hdir;
}

static int query_file_by_dir_handle(char *dir, char *file)
{
    IO_STATUS_BLOCK io;
    UNICODE_STRING file_mask;
    HANDLE hdir;
    wchar_t *wfile;
    struct {
	FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
	WCHAR namebuf[NAME_MAX + 1];
    } fdi;
    hdir = get_directory_handle(dir);
    wfile = malloc(sizeof(WCHAR)*(strlen(file)+1));
    wstr_from_str(wfile, file);
    file_mask.Buffer = wfile;
    file_mask.Length = wcslen(file_mask.Buffer) * sizeof(WCHAR);
    file_mask.MaximumLength = file_mask.Length + sizeof(WCHAR);
    if (NtQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &fdi, sizeof(fdi),
	FileIdBothDirectoryInformation, 1, &file_mask, 1))
    {
	do_err("NtQueryDirectoryFile failed (%s)\n", file);
    }
    NtClose(hdir);
    return 0;
}

static int query_file_by_handle(char *file)
{
    OBJECT_ATTRIBUTES attr;
    UNICODE_STRING ufile;
    struct {
	FILE_ALL_INFORMATION fai;
	WCHAR namebuf[NAME_MAX + 1];
    } fai;
    wchar_t *wfile;
    char *p, win_file[512];
    IO_STATUS_BLOCK io;
    HANDLE h;
    sprintf(win_file, "\\??\\c:/cygwin%s", file);
    for (p = win_file; *p; p++)
    {
	if (*p=='/')
	    *p = '\\';
    }
    wfile = malloc(sizeof(WCHAR)*(strlen(win_file)+1));
    wstr_from_str(wfile, win_file);
    ufile.Buffer = wfile;
    ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
    ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
    InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
    if (NtCreateFile(&h, SYNCHRONIZE | GENERIC_READ, &attr, &io, NULL, 0,
	FILE_SHARE_VALID_FLAGS, FILE_OPEN,
	FILE_OPEN_REPARSE_POINT|FILE_OPEN_FOR_BACKUP_INTENT, NULL, 0))
    {
	do_err("NtCreateFile failed (%s)\n", file);
    }
    if (NtQueryInformationFile(h, &io, &fai, sizeof(fai), 
FileAllInformation))
	do_err("NtQueryInformationFile failed (%s)\n", file);
    NtClose(h);
    return 0;
}

static int query_file_by_name(char *file)
{
    int ret;
    OBJECT_ATTRIBUTES attr;
    FILE_NETWORK_OPEN_INFORMATION fi;
    UNICODE_STRING ufile;
    wchar_t *wfile;
    char *p, win_file[512];
    sprintf(win_file, "\\??\\c:/cygwin%s", file);
    for (p = win_file; *p; p++)
    {
	if (*p=='/')
	    *p = '\\';
    }
    wfile = malloc(sizeof(WCHAR)*(strlen(win_file)+1));
    wstr_from_str(wfile, win_file);
    ufile.Buffer = wfile;
    ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
    ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
    InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
    ret = NtQueryFullAttributesFile(&attr, &fi);
    free(wfile);
    return ret;
}

static int verbose = 0;
int main(int argc, char *argv[])
{
    char *dir = "/bin", fullname[512], *method = "";
    struct dirent **namelist, *d;
    int i, n = 0, ret = -1;
    struct stat buf;
    u64 start, end;
    IO_STATUS_BLOCK io;
    HANDLE hdir;
    struct {
	FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
	WCHAR dummy_buf[NAME_MAX + 1];
    } fdi;
    argv++;
    timeBeginPeriod(1);
    usleep(100); /* it takes some time for timeBeginPeriod() to work */
    if (!*argv)
	usage();
    method = *argv++;
    if (*argv)
	dir = *argv++;
    if (*argv)
	usage();
    printf("testing %s %s ", dir, method);
    start = time_monotonic_nsec();
    if (!strcmp(method, "lstat"))
    {
	if ((n = scandir(dir, &namelist, 0, alphasort))>=0)
	    ret = 0;
    }
    else
    {
	hdir = get_directory_handle(dir);
	namelist = calloc(sizeof(*namelist), 100000);
	while (!NtQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &fdi,
	    sizeof(fdi), FileIdBothDirectoryInformation, 1, NULL, 0))
	{
	    d = namelist[n] = calloc(sizeof(**namelist), 1);
	    sprintf(d->d_name, "%.*ls",
		(int)fdi.fdi.FileNameLength/2, fdi.fdi.FileName);
	    n++;
	}
	ret = 0;
	NtClose(hdir);
    }
    if (ret)
	do_err("scandir: %s\n", strerror(errno));
    end = time_monotonic_nsec();
    printf("%d files ", n);
    if (verbose)
	printf("scandir(%gms) ", ((double)end-start)/NSEC_PER_MS);
    start = time_monotonic_nsec();
    for (i=0; i<n; i++)
    {
	d = namelist[i];
	if (!strcmp(".", d->d_name) || !strcmp("..", d->d_name))
	    continue;
	sprintf(fullname, "%s/%s", dir, d->d_name);
	if (verbose)
	    printf("%s\n", fullname);
	if (!strcmp(method, "GFA"))
	    ret = query_file_by_name(fullname);
	else if (!strcmp(method, "QIF"))
	    ret = query_file_by_handle(fullname);
	else if (!strcmp(method, "QDF"))
	    ret = query_file_by_dir_handle(dir, d->d_name);
	else if (!strcmp(method, "lstat"))
	    ret = lstat(fullname, &buf);
	else
	    do_err("unknow method %s", method);
	if (ret)
	    do_err("stat %s: %s\n", d->d_name, strerror(errno));
	free(d);
    }
    free(namelist);
    end = time_monotonic_nsec();
    printf("stat() %4.4gms, per file: %4.4gms ",
	((double)end-start)/NSEC_PER_MS,
	((double)end-start)/n/NSEC_PER_MS);
    printf("\n");
    return 0;
}

==================================
end of code

On 29/9/2010 11:08 AM, Derry Shribman wrote:
> Hi,
>
>  >> Doesn't the 'noacl' mount option provide that already?
>  >
>  > Partially, there are also the ihash and the exec/notexec options. A lot
>  > has been already discussed on the cygwin-patches list, see, for instance
>
> The problem with mount options is that they are 'static'. They require a
> cygwin 'reboot' and they do not allow 'inheritance' for subprocesses,
> and do not allow concurrent processes running in different modes.
>
> Dynamic options via CYGWIN env allow setting stuff in runtime, in
> /etc/profile, ~/bashrc, or for specific commands (and their
> subprocesses), such as:
> CYGWIN=no_nlink rsync c:/... z:/...
>
> This allows the user to be free to decide where to relax POSIX
> compliance in order to achieve speed.
>
> It also allows application developers (such as 'git'), to decide in
> their code how they want Cygwin to behave.
> In 'git' for example, it does need stat's nlink (number of hard links),
> and actually, not even n_ino (the inode number). Cygwin's git
> performance was ultra-slow, and they improved it by not using Cygwin's
> stat(), rather re-implementing their own 'quick-stat' which worked
> directly with Win32 API.
>
> If Cygwin would have supported dynamic options (as opposed to mount time
> options), instead of the large 'ifdef __CYGWIN__' code, it would simply
> be adding 'setenv("CYGWIN", "no_nlink no_inode")' to the code in git's
> main().
>
> This allow applications to declare they will never look into the
> 'st_ino' and 'st_nlink'. The authors of an application, at the time of
> writing it, know whether their code accesses these fields or not.
>
> This does not reduce the over-all correctness of the application:
> cygwin1.dll does not collect this extra expensive info - that the
> application does not use anyway.
>
> Many applications can benefit from this, and implementation wise it is
> quite simple, and actually has already been implemented in Yoni's patch!
>
> Derry Shribman
>
> On 9/29/2010 10:32 AM, Corinna Vinschen wrote:
> > On Sep 29 05:50, Andy Koppe wrote:
> > > On 28 September 2010 18:44, Warren Young wrote:
> > > > On 9/28/2010 9:10 AM, Christopher Faylor wrote:
> > > > > It isn't extremely surprising that Linux access speed for a filesystem
> > > > > in a simulated environment, which presumably does not go through
> > > > > multiple layers of DLLs, would be faster than Cygwin.
> > > >
> > > > I think it more likely that the HGFS driver doesn't try to preserve
> > > > full
> > > > POSIX semantics. There's plenty of precedent: vfat, iso9660... One
> > > > could
> > > > probably verify this faster by examining the driver's source code
> > > > (http://open-vm-tools.sourceforge.net/) than by tracing syscalls.
> > > >
> > > > If that's the explanation, it points at a possible path forward.
> > > >
> > > > On Linux, these secondary filesystems aren't expected to provide
> > > > full POSIX
> > > > semantics, simply because they are secondary. No one cries very hard
> > > > that
> > > > you can't make symlinks on a FAT-formatted USB stick.
> > > >
> > > > Yet, there's probably no technical reason you couldn't get a POSIX-like
> > > > system to run on a crippled filesystem. It's probably even been done
> > > > lots
> > > > of times before in the embedded world. Some of the PC Unix systems
> > > > from the
> > > > 80s and early 90s were pretty screwy in this way, too. Screwy doesn't
> > > > prevent you from doing useful work, though.
> > > >
> > > > Would it not be useful to have a mode in Cygwin that purposely skips
> > > > any
> > > > POSIX semantics that it can't get for free by making the POSIX syscalls
> > > > nothing more than thin wrappers around the nearest equivalent Win32
> > > > API? If
> > > > you put it in this mode and it breaks, you get to keep both pieces.
> > > > There
> > > > are those who would happily accept the speed increase for loss of some
> > > > functionality. I wouldn't, but some would. I'd bet a lot of the 3PPs
> > > > are
> > > > in that group, since they know their target environment very well.
> > >
> > > Doesn't the 'noacl' mount option provide that already?
> >
> > Partially, there are also the ihash and the exec/notexec options. A lot
> > has been already discussed on the cygwin-patches list, see, for instance
> >
> > http://cygwin.com/ml/cygwin-patches/2010-q3/msg00035.html
> > http://cygwin.com/ml/cygwin-patches/2010-q3/msg00036.html
> >
> > and
> >
> > http://cygwin.com/ml/cygwin-patches/2010-q3/msg00071.html
> > http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html
> >
> >
> > Corinna