Most people will likely not start with a computer dedicated to function solely for running a VPO software like GrandOrgue. I sure didn't. The computer I've at the moment got embedded into my organ console still does perform other tasks than to only allow me to play the organ. So feel absolutely free to use whatever computer you have to install and run GrandOrgue to test it!
However, you'll likely quickly discover that the performance power/experience is directly related to the computers capacity. A computer that allow for high quality demands in both playing experience and choice of a high end sampleset will need to have certain features that's perhaps not available in just any "off the shelf" cheap computer. This is not to say that you cannot use one though! Overall you want a computer that has enough resources to play many simultaneous samples at a low latency and memory to load the biggest sampleset you intend to use. Hard disk space is rarely a problem anymore but it's performance can be so choose one with good (low) access time and high rotation speed, if you don't go for a SSD of course. Also the audio capabilities of the computer is not the least important thing either, after all you want a good listening experience while playing!
First a choice between a laptop (mobile computer) or a stationary has to be made. From testing both I can say that you'll get much more computer power from a stationary at the same price than you get from a laptop. Also the stationary is much easier to upgrade and have less trouble with performance inhibiting features (used for energy saving, cpu restrictions and limited number of USB ports etc). The only real advantage of the laptop (or even netbook in my case) is the portability if it's needed. Otherwise I'd really recommend sticking to a stationary!
If you're not building from parts yourself then you'll have to live with what combination of hardware is in the computer package, but here I'll try to discuss the most important parts individually. First you'll have to look at what the motherboard will offer you in order to make a good choice for the foundation of the computer. It decides what kind of cpus you'll be able to use (type of socket), how much RAM (and at what speed/clocking), how many USB ports you'll have, if it has built-in audio and graphics etc. Also it's form factor can be important as the size might matter depending on your console/setup. Your exact needs might differ depending on how your MIDI equipment eventually is to be connected to the computer.
From the cpu you want processing power. The number of cores and their speed will eventually determine how high the polyphony (amount of simultaneous samples) can be used. Count the number of real cores and ignore hyperthreading if it's present. As a base line I'd suggest something at least in the area of a core i5 at the present time even though I'm myself is using an older core i3. Intel or AMD processors might be a personal choice but I prefer Intel. 64 bit capability is more common nowadays but to fully utilize it you also need more than 4 GB of RAM and a 64 bit operative system.
So, next off to the memory (RAM). If you're planning to use a 64 bit system (which I do recommend) you can go way above 4 GB. Exact maximum amount will depend on the motherboard in questing but as the prices on memory have dropped I'd recommend to get at least 8 GB (or more if you can) to have some future expansion room. If you already know what samplesets you intend to use you'll know the requirements to load it at full quality. But equally important to the amount of memory is it's speed/clocking. Get memory with the lowest latency - highest speed you can! It does pay off in terms of what latency you'll be able to achieve in playing.
Standard hard drive or SSD? Well, it depends. Not all SSDs are equal, some perform even worse than a traditional rotating disc hard drive so be sure to read up on it's performance if you choose to go that road. The benefit of the SSD is often much quicker startups and quick access times at the cost of less amount of storage space for the price of a normal hard drive. Today I'd go with a good SSD of moderate size (say from around 120 GB and up), no need to get the biggest ones as the VPO and OS won't really need that much space.
Soundcard? I'd recommend to stick with either the built-in or an internal pci version for good latency. Firewire have good performance reputation but I've never used it and unfortunately it apparently can be a bit problematic to configure also. USB audio interfaces might work well for simple recording but I wouldn't trust it for low latency VPO playing. Advantage of built-in versions are support of many samplerates and often high amount of channels available for output. General support of them also is good, be sure to check any pci card (or firewire if you're into them) for it's reported functionality (especially if you intend to use Linux as OS). High end soundcards (pci or firewire) should in theory be better, but if you'll notice it depends on other links in your audio chain (amplifier and speaker quality). Also you'll have to weigh in the support factor and possibility of multi channel usage if you want it.
Dedicated graphics card? It's not absolutely necessary if motherboard and cpu can deliver the feature since the VPO usage is not really so graphic intense. But certain cards can improve overall performance and relieve the cpu which might be beneficial. Also it might be interesting if you plan to use multiple monitors.
You generally don't need any cd/dvd/blue-ray player/writer in a pure VPO computer and in fact it might be better the less amounts of parts the computer is made up of. However, some will still find such things highly usable. Be sure to have a current supply that's ample for the need, err on the safe side and have too high capacity if in doubt.
Overall noise level! An ideal VPO computer is silent! Some fans might be needed but be sure to have the most silent ones you can afford and have a chassi that help with the cooling. The noise level is yet another advantage of using SSD over traditional hard drive as it doesn't contain moving parts. The actual placement of the computer will affect its audability but it usually still must get fresh cool air.
Finally, what OS to use? Well, if you ask me I'll recommend a 64 bit Linux distribution of course. I've more or less stopped using Windows for anything serious since 2007 and I don't regret it. Ubuntu is a very nice distribution even though some might find it a bit bloated it's fairly easy to use for anyone and has a massive amount of software to easily install through the package manager. With some tweaking I think Ubuntu works just fine. There are some other derivates like XUbuntu and LUbuntu that are promising too. Debian is the basis for Ubuntu but is also a bit more difficult to start with and might have more problems with hardware support as it's policies for inclusion of drivers are very strict. Of course there are many possible Linux distributions that will work fine, OpenSUSE is one of them as is Fedora, but so far I've found Ubuntu to suite me the best.