tuning a VERY heavily (30.0) loaded server

Path: supernews.google.com!sn-xit-03!supernews.com!hermes.visi.com!news-out.visi.com!newsxfer.interpacket.net!newsfeed.cwix.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 11:11:44 -0600
From: "Michael C . Wu" <keic...@iteration.net>
To: dil...@freebsd.org, g...@freebsd.org, f...@freebsd.org, hack...@freebsd.org
Subject: tuning a VERY heavily (30.0) loaded server
Original-Message-ID: <20010320111144.A51924@peorth.iteration.net>
Reply-To: "Michael C . Wu" <keic...@peorth.iteration.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 17:12:16 GMT
Message-ID: <fa.kbofbpv.17hi9io@ifi.uio.no>
Lines: 86

[Lengthy email, bear with me please, it is quite interesting.
This box averages 30.0 load with no problems.]

system stats at 
http://zoo.ee.ntu.edu.tw/~keichii/

Hello Everyone,

I have a friend who admins a very heavily loaded BBS server.
(In Taiwan, BBS'es are still very popular, because they 
are the primary form of scholastic communication in colleges/universities.
And FreeBSD runs on most of the university systems in Taiwan ;) )

This box is rather a FreeBSD advocacate itself, as you will see why.

It runs an self-wrote PERL SMTP daemon. (Sendmail and Postfix croaks)
SMTPD pipes the mail to "bbsmail" that delivers the mail to 
BBS users.  SMTPd averages about 

BBSd averages about 3000 users at any given time of the day,
Peak usage is about 4300 users before the box dies.  
Each user averages 4-5KB/sec bandwidth.
BBSd is an in-house modification of a popular BBSD in Taiwan.

There is an innd backend to BBSd that gets a full feed of tw.bbs.*
and many other local newsgroups.

Average file size is about 4K.  /home/bbsusers* is on a vinum
stripe'd volume with 3 Ultra160 9G 10000RPM drives on sym0 at stripe
size 256K,  Greg: I know this should be a prime number,
can we safely use <150K stripe sizes? CPU time is not a problem.

The other parts of the system rest on 3*Ultra160 9g 10K RPM on AHC0
at stripe size 256K.

Physical memory is 2.5 GB.  We do MFS and it croaks/crashes
at midnight, our peak load time.  We do md0, it croaks before
peak time.

Dual PIII-750 CPU's

Due to the structure of BBS's, we cannot split the load across
different servers.  We also think that we probably cannot
get more performance out of hardware upgrades that we can afford.
(i.e. Please don't tell us to buy a Starfire 4500 :-) We are all volunteer
werkers at El Cheapo university budgets.)

We average around 30.0 server load with no noticeable delays
for users.  Peak load is up to 50.0.  Average process count
is around 4000 to 5000.

We have followed Alfred's advice to do sysctl -w vfs.vmioenable=1
It allows us to survive the peak load a little longer than before.
And we are putting our logs of sockstat, iostat 5, vmstat 5,
netstat 5, dmesg, uname -a on the following URL.

http://zoo.ee.ntu.edu.tw/~keichii/

*DRUM ROLL*
What do you think we can do to make this server survive the 
  peak load of around 5000 users? :)

* How should we setup our IPFW?  

* What should be the optimal newfs and tunefs 
  configurations for our filesystems?

* What should we try as vinum stripe sizes?

* What is possibly the bottleneck that we have for load 30.0?
  (since we are not CPU-bound nor memory bound)

* Is there any VM tweaks that we can do?

* Anything else we can do?

Thanks,
Michael
-- 
+-----------------------------------------------------------+
| keic...@iteration.net         | keic...@freebsd.org       |
| http://iteration.net/~keichii | Yes, BSD is a conspiracy. |
+-----------------------------------------------------------+

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-03!supernews.com!cyclone-sf.pbi.net!64.42.15.2!HSNX.atgi.net!feeder.via.net!diablo.netcom.net.uk!netcom.net.uk!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 09:27:17 -0800
From: Alfred Perlstein <bri...@wintelcom.net>
To: "Michael C . Wu" <keic...@iteration.net>
Cc: dil...@freebsd.org, g...@freebsd.org, f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded server
Original-Message-ID: <20010320092717.R29888@fw.wintelcom.net>
Original-References: <20010320111144.A51...@peorth.iteration.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20010320111144.A51924@peorth.iteration.net>; from keichii@iteration.net on Tue, Mar 20, 2001 at 11:11:44AM -0600
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 17:27:46 GMT
Message-ID: <fa.ijji7fv.kn2gps@ifi.uio.no>
References: <fa.kbofbpv.17hi9io@ifi.uio.no>
Lines: 62

* Michael C . Wu <keic...@iteration.net> [010320 09:11] wrote:
> [Lengthy email, bear with me please, it is quite interesting.
> This box averages 30.0 load with no problems.]

cool..

> system stats at 
> http://zoo.ee.ntu.edu.tw/~keichii/

Where's the crashdump/traceback?

> Physical memory is 2.5 GB.  We do MFS and it croaks/crashes
> at midnight, our peak load time.  We do md0, it croaks before
> peak time.

Explain the crash.  What is md0/MFS being used for?  Why do you
need it?

> Due to the structure of BBS's, we cannot split the load across
> different servers.  We also think that we probably cannot
> get more performance out of hardware upgrades that we can afford.
> (i.e. Please don't tell us to buy a Starfire 4500 :-) We are all volunteer
> werkers at El Cheapo university budgets.)

Well, getting hardware RAID is always a nice thing and really not
too expensive.

> We have followed Alfred's advice to do sysctl -w vfs.vmioenable=1
> It allows us to survive the peak load a little longer than before.

cool..

> And we are putting our logs of sockstat, iostat 5, vmstat 5,
> netstat 5, dmesg, uname -a on the following URL.
> 
> http://zoo.ee.ntu.edu.tw/~keichii/
> 
> *DRUM ROLL*
> What do you think we can do to make this server survive the 
>   peak load of around 5000 users? :)
> 

[snip several non-interesting ideas]

> * Anything else we can do?

Well first off, telling us which version of FreeBSD this is...

Second, provide a crashdump with debug symbols, and show us
the backtrace.

Third, consider alternatives to MFS since it seems to be a key
factor in your stability problems.  If you just need a pretty
fast /tmp, I would use a softupdates partition as it's probably
more effecient than MFS/MD.

-- 
-Alfred Perlstein - [bri...@wintelcom.net|alf...@freebsd.org]

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-02!supernews.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 11:38:18 -0600
From: "Michael C . Wu" <keic...@iteration.net>
To: Alfred Perlstein <bri...@wintelcom.net>
Cc: "Michael C . Wu" <keic...@iteration.net>, dil...@freebsd.org,
        g...@freebsd.org, f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded server
Original-Message-ID: <20010320113818.B52586@peorth.iteration.net>
Reply-To: "Michael C . Wu" <keic...@peorth.iteration.net>
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20010320092717.R29888@fw.wintelcom.net>; from bright@wintelcom.net on Tue, Mar 20, 2001 at 09:27:17AM -0800
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 17:38:55 GMT
Message-ID: <fa.kb8pb9v.141ca2o@ifi.uio.no>
References: <fa.ijji7fv.kn2gps@ifi.uio.no>
Lines: 82

On Tue, Mar 20, 2001 at 09:27:17AM -0800, Alfred Perlstein scribbled:
| * Michael C . Wu <keic...@iteration.net> [010320 09:11] wrote:
| > [Lengthy email, bear with me please, it is quite interesting.
| > This box averages 30.0 load with no problems.]
| 
| cool..
| 

FreeBSD zoo.ee.ntu.edu.tw 4.2-STABLE FreeBSD 4.2-STABLE 
#0: Tue Mar 20 11:10:46 CST 2001     root@:/usr/src/sys/compile/SimFarm  i386

| > system stats at 
| > http://zoo.ee.ntu.edu.tw/~keichii/
| 
| Where's the crashdump/traceback?

I'll try to get one tomorrow night, It always crashes. :)
It's quite hard trying to get a crashdump when you are 15000miles
away from the console.

| > Physical memory is 2.5 GB.  We do MFS and it croaks/crashes
| > at midnight, our peak load time.  We do md0, it croaks before
| > peak time.
| 
| Explain the crash.  What is md0/MFS being used for?  Why do you
| need it?

md0/MFS is used for caching the articles that BBS users read.
They often read the same articles over and over again,
and we find that a 128MB MFS/md0 will have 70% hitrate

When our MFS/md0 fills up after long usage, the box easily
dies.  (We crontab clean the mfs, but sometimes the load
shoots up for no reason and is not able to clean the mfs in time.)
If we dont do this cache, the data for the bulletin boards

| > Due to the structure of BBS's, we cannot split the load across
| > different servers.  We also think that we probably cannot
| > get more performance out of hardware upgrades that we can afford.
| > (i.e. Please don't tell us to buy a Starfire 4500 :-) We are all volunteer
| > werkers at El Cheapo university budgets.)
| 
| Well, getting hardware RAID is always a nice thing and really not
| too expensive.

I looked into that, it seems that hardware RAID will have less performance
due to hw raid cards' onboard CPU bounding it.

| > We have followed Alfred's advice to do sysctl -w vfs.vmioenable=1
| > It allows us to survive the peak load a little longer than before.
| 
| cool..
| 
| > And we are putting our logs of sockstat, iostat 5, vmstat 5,
| > netstat 5, dmesg, uname -a on the following URL.
| > 
| > http://zoo.ee.ntu.edu.tw/~keichii/
| > * Anything else we can do?
| 
| Well first off, telling us which version of FreeBSD this is...
| 
FreeBSD zoo.ee.ntu.edu.tw 4.2-STABLE FreeBSD 
4.2-STABLE #0: Tue Mar 20 11:10:46 CST 2001     
root@:/usr/src/sys/compile/SimFarm  i386

| Second, provide a crashdump with debug symbols, and show us
| the backtrace.
| 
| Third, consider alternatives to MFS since it seems to be a key
| factor in your stability problems.  If you just need a pretty
| fast /tmp, I would use a softupdates partition as it's probably
| more effecient than MFS/MD.

The harddrives will die very quickly if we don't have MFS...
-- 
+-----------------------------------------------------------+
| keic...@iteration.net         | keic...@freebsd.org       |
| http://iteration.net/~keichii | Yes, BSD is a conspiracy. |
+-----------------------------------------------------------+

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-02!supernews.com!nntp-relay.ihug.net!ihug.co.nz!news-spur1.maxwell.syr.edu!news.maxwell.syr.edu!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 09:50:51 -0800 (PST)
From: Matt Dillon <dil...@earth.backplane.com>
Original-Message-Id: <200103201750.f2KHopk94248@earth.backplane.com>
To: "Michael C . Wu" <keic...@iteration.net>
Cc: Alfred Perlstein <bri...@wintelcom.net>,
        "Michael C . Wu" <keic...@iteration.net>, g...@freebsd.org,
        f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded server
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net> <20010320113818.B52...@peorth.iteration.net>
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 17:52:02 GMT
Message-ID: <fa.j0gr0iv.p3cnrg@ifi.uio.no>
References: <fa.kb8pb9v.141ca2o@ifi.uio.no>
Lines: 26

    One thing that comes to mind is that you can smarthost your outgoing
    email to another host so the queues don't build up.  This should
    greatly reduce mail load.  In fact, I would recommend offloading email
    entirely if possible... email always hits disks hard.

    Definitely get rid of MFS.  MFS wastes 2x the memory allocated to it.
    Use a softupdates-enabled filesystem in place of MFS, or use a 
    swap-backed VN-based partition with softupdates enabled.

    Alfred's vmiodirenable suggestion is a good one.

    With all the memory you have you can also try turning off write_behind,
    e.g. setting vfs.write_behind to 0.

    I don't have enough information on the type of paging your machine
    is doing or the disk configuration.  If you have multiple HD's, swap
    should definitely be spread across at least two of them.

    A few minutes worth of 'vmstat 1' output during the heavily loaded 
    period would be useful, plus 'sysctl -a | fgrep vm'.  I might be able
    to make suggestions on optimizing the VM system.

						-Matt

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-02!supernews.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 09:54:55 -0800 (PST)
From: Matt Dillon <dil...@earth.backplane.com>
Original-Message-Id: <200103201754.f2KHstW94364@earth.backplane.com>
To: "Michael C . Wu" <keic...@iteration.net>
Cc: Alfred Perlstein <bri...@wintelcom.net>,
        "Michael C . Wu" <keic...@iteration.net>, g...@freebsd.org,
        f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded server
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net> <20010320113818.B52...@peorth.iteration.net>
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 17:55:47 GMT
Message-ID: <fa.iuh31qv.v2kmia@ifi.uio.no>
References: <fa.kb8pb9v.141ca2o@ifi.uio.no>
Lines: 22

:md0/MFS is used for caching the articles that BBS users read.
:They often read the same articles over and over again,
:and we find that a 128MB MFS/md0 will have 70% hitrate
:
:When our MFS/md0 fills up after long usage, the box easily
:dies.  (We crontab clean the mfs, but sometimes the load
:shoots up for no reason and is not able to clean the mfs in time.)
:If we dont do this cache, the data for the bulletin boards

    Definitely throw away MFS.  A normal filesystem is plenty good enough
    for caching articles that BBS users read.  MFS will just waste memory
    unnecessarily.

    It does seem to me that you might not have sufficient swap configured
    either, as per Ted's thought.  With 2.5G of physical memory, You 
    should have *AT LEAST* 3G of configured swap.  I would recommend a 1G
    swap partition on each of your three 9G drives (for 3G total).

						-Matt

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-03!supernews.com!news-out.usenetserver.com!news-out-sjo.usenetserver.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 12:01:12 -0600
From: "Michael C . Wu" <keic...@iteration.net>
To: iz...@ms26.hinet.net, cr...@math.psu.edu
Cc: Alfred Perlstein <bri...@wintelcom.net>,
        "Michael C . Wu" <keic...@iteration.net>, dil...@freebsd.org,
        g...@freebsd.org, f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded scerver
Original-Message-ID: <20010320120112.C52586@peorth.iteration.net>
Reply-To: "Michael C . Wu" <keic...@peorth.iteration.net>
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net> <20010320113818.B52...@peorth.iteration.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20010320113818.B52586@peorth.iteration.net>; from keichii@iteration.net on Tue, Mar 20, 2001 at 11:38:18AM -0600
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 18:02:53 GMT
Message-ID: <fa.kb8fahv.141uaqg@ifi.uio.no>
References: <fa.kb8pb9v.141ca2o@ifi.uio.no>
Lines: 36

MRTG Graph at
http://zoonews.ee.ntu.edu.tw/mrtg/zoo.html

| 
| FreeBSD zoo.ee.ntu.edu.tw 4.2-STABLE FreeBSD 4.2-STABLE 
| #0: Tue Mar 20 11:10:46 CST 2001     root@:/usr/src/sys/compile/SimFarm  i386
| 
| | > system stats at 
| | > http://zoo.ee.ntu.edu.tw/~keichii/
| md0/MFS is used for caching the articles that BBS users read.
| They often read the same articles over and over again,
| and we find that a 128MB MFS/md0 will have 70% hitrate
| 
| When our MFS/md0 fills up after long usage, the box easily
| dies.  (We crontab clean the mfs, but sometimes the load
| shoots up for no reason and is not able to clean the mfs in time.)
| If we dont do this cache, the data for the bulletin boards
| 

Another problem is that we have around 4000+ processes accessing
lots of SHM at the same time..

The *UGLY* source code for the BBS is at 
http://zoo.ee.ntu.edu.tw/~keichii/zoo_bbsd_src.tgz

We can only provide crash dumps for trusted people because
of the thousands of passwords in the dump.

-- 
+-----------------------------------------------------------+
| keic...@iteration.net         | keic...@freebsd.org       |
| http://iteration.net/~keichii | Yes, BSD is a conspiracy. |
+-----------------------------------------------------------+

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-02!supernews.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 10:09:09 -0800
From: Alfred Perlstein <bri...@wintelcom.net>
To: "Michael C . Wu" <keic...@peorth.iteration.net>
Cc: iz...@ms26.hinet.net, cr...@math.psu.edu,
        "Michael C . Wu" <keic...@iteration.net>, dil...@freebsd.org,
        g...@freebsd.org, f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded scerver
Original-Message-ID: <20010320100909.T29888@fw.wintelcom.net>
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net> <20010320113818.B52...@peorth.iteration.net> <20010320120112.C52...@peorth.iteration.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <20010320120112.C52586@peorth.iteration.net>; from keichii@iteration.net on Tue, Mar 20, 2001 at 12:01:12PM -0600
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 18:12:21 GMT
Message-ID: <fa.ijj46vv.kncg9k@ifi.uio.no>
References: <fa.kb8fahv.141uaqg@ifi.uio.no>
Lines: 56

* Michael C . Wu <keic...@iteration.net> [010320 10:01] wrote:
> MRTG Graph at
> http://zoonews.ee.ntu.edu.tw/mrtg/zoo.html
> 
> | 
> | FreeBSD zoo.ee.ntu.edu.tw 4.2-STABLE FreeBSD 4.2-STABLE 
> | #0: Tue Mar 20 11:10:46 CST 2001     root@:/usr/src/sys/compile/SimFarm  i386
> | 
> | | > system stats at 
> | | > http://zoo.ee.ntu.edu.tw/~keichii/
> | md0/MFS is used for caching the articles that BBS users read.
> | They often read the same articles over and over again,
> | and we find that a 128MB MFS/md0 will have 70% hitrate
> | 
> | When our MFS/md0 fills up after long usage, the box easily
> | dies.  (We crontab clean the mfs, but sometimes the load
> | shoots up for no reason and is not able to clean the mfs in time.)
> | If we dont do this cache, the data for the bulletin boards
> | 
> 
> Another problem is that we have around 4000+ processes accessing
> lots of SHM at the same time..

How much SHM?  Like, what's the combined size of all segments in
the system?  You can make SHM non-pageable which results in a lot
of saved memory for attached processes.

You want to be after this date and have this file:


Revision 1.3.2.3 / (download) - annotate - [select for diffs], Sun Dec 17 02:05:41 2000 UTC (3 months ago) by alfred 
Branch: RELENG_4 
Changes since 1.3.2.2: +37 -32 lines
Diff to previous 1.3.2.2 (colored) to branchpoint 1.3 (colored) next main 1.4 (colored)

MFC: phys_pager fix for multiple segments

Then set kern.ipc.shm_use_phys=1

> The *UGLY* source code for the BBS is at 
> http://zoo.ee.ntu.edu.tw/~keichii/zoo_bbsd_src.tgz

tis ok, maybe later... though :)

> 
> We can only provide crash dumps for trusted people because
> of the thousands of passwords in the dump.

Heh. :)

-- 
-Alfred Perlstein - [bri...@wintelcom.net|alf...@freebsd.org]


To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-02!supernews.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 12:22:45 -0600
From: "Michael C . Wu" <keic...@iteration.net>
To: Matt Dillon <dil...@earth.backplane.com>
Cc: "Michael C . Wu" <keic...@iteration.net>,
        Alfred Perlstein <bri...@wintelcom.net>, g...@freebsd.org,
        f...@freebsd.org, hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded server
Original-Message-ID: <20010320122245.E52586@peorth.iteration.net>
Reply-To: "Michael C . Wu" <keic...@peorth.iteration.net>
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net> <20010320113818.B52...@peorth.iteration.net> <200103201750.f2KHopk94...@earth.backplane.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.2.5i
In-Reply-To: <200103201750.f2KHopk94248@earth.backplane.com>; from dillon@earth.backplane.com on Tue, Mar 20, 2001 at 09:50:51AM -0800
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 18:26:46 GMT
Message-ID: <fa.kcolb1v.16hgaal@ifi.uio.no>
References: <fa.j0gr0iv.p3cnrg@ifi.uio.no>
Lines: 42

On Tue, Mar 20, 2001 at 09:50:51AM -0800, Matt Dillon scribbled:
|     One thing that comes to mind is that you can smarthost your outgoing
|     email to another host so the queues don't build up.  This should
|     greatly reduce mail load.  In fact, I would recommend offloading email
|     entirely if possible... email always hits disks hard.
| 
|     Definitely get rid of MFS.  MFS wastes 2x the memory allocated to it.
|     Use a softupdates-enabled filesystem in place of MFS, or use a 
|     swap-backed VN-based partition with softupdates enabled.

|     Alfred's vmiodirenable suggestion is a good one.
| 
|     With all the memory you have you can also try turning off write_behind,
|     e.g. setting vfs.write_behind to 0.

done. :) Thank you
|     I don't have enough information on the type of paging your machine
|     is doing or the disk configuration.  If you have multiple HD's, swap
|     should definitely be spread across at least two of them.
| 
|     A few minutes worth of 'vmstat 1' output during the heavily loaded 
|     period would be useful, plus 'sysctl -a | fgrep vm'.  I might be able

sysctl -a always crashes the system.  It happens on other similiarly
loaded BBS'es in Taiwan.

|     to make suggestions on optimizing the VM system.

We have 'vmstat 5' available at http://zoo.ee.ntu.edu.tw/~keichii/
Fresh hot vmstat 1 log at
http://zoo.ee.ntu.edu.tw/~keichii/vmstat_1.log

-- 
+-----------------------------------------------------------+
| keic...@iteration.net         | keic...@freebsd.org       |
| http://iteration.net/~keichii | Yes, BSD is a conspiracy. |
+-----------------------------------------------------------+

To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-03!supernews.com!newsswitch.lcs.mit.edu!sunqbc.risq.qc.ca!newsfeeds.belnet.be!news.belnet.be!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Original-Date: Tue, 20 Mar 2001 10:48:45 -0800 (PST)
From: Matt Dillon <dil...@earth.backplane.com>
Original-Message-Id: <200103201848.f2KImj995560@earth.backplane.com>
To: "Michael C . Wu" <keic...@iteration.net>
Cc: Alfred Perlstein <bri...@wintelcom.net>, g...@freebsd.org, f...@freebsd.org,
        hack...@freebsd.org
Subject: Re: tuning a VERY heavily (30.0) loaded server
Original-References: <20010320111144.A51...@peorth.iteration.net> <20010320092717.R29...@fw.wintelcom.net> <20010320113818.B52...@peorth.iteration.net> <200103201750.f2KHopk94...@earth.backplane.com> <20010320122245.E52...@peorth.iteration.net>
Sender: owner-freebsd-hack...@freebsd.org
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 18:49:26 GMT
Message-ID: <fa.it0qvhv.tjkgp8@ifi.uio.no>
References: <fa.kcolb1v.16hgaal@ifi.uio.no>
Lines: 33

:We have 'vmstat 5' available at http://zoo.ee.ntu.edu.tw/~keichii/
:Fresh hot vmstat 1 log at
:http://zoo.ee.ntu.edu.tw/~keichii/vmstat_1.log
:
:-- 
:+-----------------------------------------------------------+
:| keic...@iteration.net         | keic...@freebsd.org       |

    Your vmstat output indicates:

    * That you have plenty of cpu
    * That you are not paging heavily (good!)

    Ah.  Your kernel config is in that directory too.  Cool.
    Looks about what I expected.  The default VM_KMEM_SIZE_MAX
    is 200MB, I'm not sure why you are reducing it to 192MB
    (but it wouldn't make much of a different I guess).

    I usually don't increase 'maxusers' above 256 myself, but
    512 should be fine.  Everything else looks fine too.

    The iostat output sheds more light on the disk activity.
    It doesn't look all that bad.  If your users are accessing
    a lot of different files it might be beneficial to mount the
    filesystems in question with the 'noatime' option.  This
    coupled with softupdates should remove any need for MFS.

					-Matt



To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message

Path: supernews.google.com!sn-xit-03!supernews.com!news-out.usenetserver.com!news-out-sjo.usenetserver.com!news.tele.dk!129.240.148.23!uio.no!nntp.uio.no!ifi.uio.no!internet-mailinglist
Newsgroups: fa.freebsd.hackers
Return-Path: <owner-freebsd-hack...@freebsd.org>
Delivered-To: freebsd-hack...@freebsd.org
Original-Date: Tue, 20 Mar 2001 22:00:40 +0100 (CET)
From: Paul Herman <pher...@frenchfries.net>
To: Matt Dillon <dil...@earth.backplane.com>
Cc: "Michael C . Wu" <keic...@iteration.net>,
        Alfred Perlstein <bri...@wintelcom.net>, <hack...@freebsd.org>
Subject: Re: tuning a VERY heavily (30.0) loaded server
In-Reply-To: <200103201848.f2KImj995560@earth.backplane.com>
Original-Message-ID: <Pine.BSF.4.33.0103202155310.14957-100000@husten.security.at12.de>
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII
Sender: owner-freebsd-hack...@freebsd.org
X-Loop: FreeBSD.ORG
Precedence: bulk
Organization: Internet mailing list
Date: Tue, 20 Mar 2001 21:35:43 GMT
Message-ID: <fa.n0o3psv.1ih2g1k@ifi.uio.no>
References: <fa.it0qvhv.tjkgp8@ifi.uio.no>
Lines: 20

On Tue, 20 Mar 2001, Matt Dillon wrote:

> :We have 'vmstat 5' available at http://zoo.ee.ntu.edu.tw/~keichii/
> :Fresh hot vmstat 1 log at
> :http://zoo.ee.ntu.edu.tw/~keichii/vmstat_1.log
>
>     I usually don't increase 'maxusers' above 256 myself, but
>     512 should be fine.  Everything else looks fine too.

I ran a heavily loaded (3.4-RELEASE) www/mysql server once which
randomly froze about every three days.  It had maxusers 512, and
backing it off to about 490 got rid of the freezing.  IIRC, it used to
not be safe to run maxusers 512, but things may have changed since
then.

-Paul.


To Unsubscribe: send mail to majord...@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message