First, it is better if you have a tool to generate a map to run it n
times locally on n hosts and then send those results out to be merged
together. The archie equivalent of this is the "ls-lR.Z" file you see
in FTP sites.
Second if you do this well it will be popular and perhaps so popular
that your server will explode from overuse.
It would be better to move slowly and go for enough funding for
infrastructure so that it's not just a gee-can-we-do-it hack that ends
up being instantly oversubscribed....
--Ed