Serious performance problems (malloc related?)

[Andy Ross <andy at plausible dot org>]

At FlightGear, we have a bunch of users building on Cygwin as their
environment (MSVC and mingw builds are possible but non-trivial, and
the configure scripts work out-of-the-box on cygwin) and complaints of
performance problems have been persistent.  When it was recently
pointed out that the simulator starup on Cygwin was 10-20 times slower
(10 seconds vs. several minutes!) than under Linux or a MSVC build, I
had to take a look. :)

Try the attached C++ file.  It's a pared down version of one of the
initialization steps we do at startup (reading a giant database of
airports, although that's not particularly important).  All it does is
read the file line-by-line and run it through a split-on-whitespace
function.  It takes *vastly* longer to complete when linked against
cygwin.dll than when compiled against the native C library with
-mno-cygwin.

If you watch the process with strace (the second field shows
microseconds since the last line), you will see that most I/O reads
complete in just a few tens of microseconds.  But every N reads*,
there is a rogue, inexplicable delay of several milliseconds.  Since
only a few blocks have completed by the time they happen, these delays
represent maybe 95% of the execution time, which happens to match the
cygwin.dll performance overhead very closely.

* N == 4 on my machine running this test code, but ~10 running the
  actual FlightGear binary.

My guess, having zero knowlege of cygwin.dll (remember I'm a linux
guy), is that something in the malloc implementation is spinning
needlessly on a synchronization primitive, or something of the sort.

Note that the problem is not I/O overhead -- if you remove the split()
call, the program completes very quickly.  It seems to be something
related to the allocation done in split().

Anyway, any help would be appreciated.  This really is a rather
serious problem.  We have some users waiting three minutes for the sim
to start.  While I would obviously prefer they upgrade to Linux to
avoid this problem, in the real world I suspect we're likely to be
losing users instead.

Thanks,
Andy

#include <ctype.h>
#include <vector>
#include <string>

// Reads and splits the input file on whitespace.  Sample input file:
//
// perl -e 'for($i=0; $i<160000; $i++) { print "AAAAA " x 12, "\n" }' \
//    > cygspd.dat
//
// $ g++ -O2 -o cygspd cygspd.cc
// $ time ./cygspd cygspd.dat
//
// real    0m26.545s
// user    0m26.468s
// sys     0m0.031s
// 
// g++ -mno-cygwin -O2 -o cygspd cygspd.cc
// time ./cygspd cygspd.dat
//
// real    0m1.689s
// user    0m0.015s
// sys     0m0.015s
//
// VERY bad performance characteristics when linked against
// cygwin.dll.  Watch under strace, and note that every N reads, there
// is an inexplicable delay of several milliseconds.  Spinning in the
// malloc implementation?

using namespace std;

vector<string> split(const string& str)
{
    vector<string> result;
    string::size_type i=0, j, len=str.length();
    while(i < len) {
        while(i < len && isspace(str[i])) ++i;
        j = i;
        while (i < len && !isspace(str[i])) ++i;
        if(j < i) {
            result.push_back(str.substr(j, i-j));
            while(i < len && isspace(str[i])) ++i;
        }
    }
    return result;
}

#define BUFSZ 2048
char buf[BUFSZ];

int main(int argc, char** argv)
{
    FILE* input = argc > 1 ? fopen(argv[1], "r") : stdin;
    vector<string> toks;
    while(fgets(buf, BUFSZ, input))
        toks = split(buf);
}

--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/