EQ: CPU, Memory usage... why so much?

[Sanna]

Ok, of course my Computer Engineering BF would think of this, and of course the Creative Writing major me has no answer.


When running EQ on a WinXP platform, you can CtrlAltDel and look at what your computer is running. The second tab tells CPU usage and Memory usage. EQ takes 99pct of the CPU usage, even when doing nothing more 'stressful' than sitting in the bazaar in trader-mode. What does it do? Why does it take so much stuff?


Plus, he has lots of other questions about coding and why it's so laggy and something related to 'why don't they use hardware acceleration' and other terms that I don't understand ^_^. Anyone have any answers I can give him? I'm not computer illiterate, especially by today's standards, but I do know there is a lot I don't know ^_^.

Thanks muchly.

-- Sanna

[Lilosh]

EQ is a resource hog.

It expands to fill whatever resources are available on the system, as a result of the way the devs coded the windowed mode.



Now, from what I've been told, EQW doesn't do that. But I can't confirm that, since I haven't tried it.


-Lilosh

[KyrosKrane]

At the simplest level, EQ is badly coded for the modern era. It was designed many years ago, using assumptions that differ drastically from modern standards.

When EQ first came out, up until a few months ago, it was the only application you were ALLOWED to run. If you ever tried to run anything else, it would immediately exit. The reasons (and the logic behind this choice, if any) are debatable, but the core issue is that the EQ client expects to be the only application that needs system resources.

Under Windows 95 and 98 (the main platforms for which EQ was designed), a program effectively has total control of the system until it says otherwise. Think of it this way. Windows has a list of running programs in its memory. It goes down the list, telling each program, "OK, you have access to system resources now. Let me know when you're done." (This is called gaining priority.) The program goes off, does its merry thing, then eventually tells Windows, "OK, I'm done now. You can let other programs have a turn." (This is called passing priority.) Windows then goes to the next program in the list and gives it priority, and so on.

This system is called cooperative multi-tasking. You rely on each program to voluntarily give up control when it no longer needs system resources, or at least every once in a while. If the programs are well designed, each of these turns lasts for a few milliseconds at most, so it seems like all the programs are running at the same time. If a program is badly written, however, it can totally lock up the system by never passing priority.

EQ, however, didn't like to share. When it gets that priority, it never passes it back to Windows to let other programs run. It used to be that when it lost priority, it would self-exit.

Enter the modern era of computing, with Windows 2000 and XP. These operating systems are far more stable, among other reasons, because they ditched cooperative multi-tasking. Instead, they rely on what's called preemptive multi-tasking. Basically, Windows tells each program, "You have up to X milliseconds to do your stuff, no more. Let me know if you finish before that." At the end of those X milliseconds, Windows forcefully grabs priority away from the program and gives it to the next one in line. This ensures that all programs get a fair share of the system resources. It also prevents one badly-written application from freezing your system.

OK, so. That's a primer on how operating systems work. Let's take a look at what EQ does. Given that I'm not a developer working on EQ, this is a simplified version of how I would write the program.

1) Check for system messages from Windows or other applications. (This is stuff like "The user clicked the mouse.")

2) Check for messages from the server for things EQ needs to deal with.

3) Process any new events and display any new changes on the screen.

4) Go to Step 1.

Grossly oversimplified, but it gets the point across. EQ does what's called a "tight loop;" it never yields priority to the operating system to do other stuff. Various hacker-type people who know a lot more about decompiling programs than I do have figured out how the EQ code was written, and identified a very specific problem that causes this 100% CPU utilization. They have posted on a number of message boards about how Sony can literally fix this problem with one line of code. The changes to the above system would be:

4) Let Windows do other stuff.

5) Go to Step 1.

They even made a little program (that doesn't work after the most recent patches, unfortunately) that simply inserts this line into EQ's code when it's loaded into memory. If you use that program, you'll see EQ's CPU utilization drop from 99% to maybe 40% or 50%.

And if you're wondering, yes, the problem really can be solved by adding a single line of code to EQ, without affecting anything else in the game or the client. Heck, those hacker-types I mentioned earlier even sent a message to Sony with the exact line of code needed and the specific spot in the source code that it should be inserted!

We can only hope that Sony doesn't disregard the message because of the messenger.

[Sanna]

wow. /agree kryos. A combination of 'don't shoot the messenger' combined with 'break the too-rigid rules to make the system better'. /crosses fingers.

/realizes it's futile.

/keeps fingers crossed.

Thanks. I've sent this link to him, maybe it'll help.


-- Sanna
I still say we petition for %C for con-color. That way I can hotkey this:
/g I spy --[ %T ]-- and %S is %C to me!



edit.
well, go me, I sent this link to him and it turns out 'he knows all this already', and what he was really asking me was 'why soe doesn't fix it already.' which of course no one knows, because soe never tells things like that. so, i go to put out a fire of my own making. /sigh. thanks for the answers tho. ^_^

[Ngreth Thergn]

Also.

As for Hardware Accelleration.

EQ came out before video cards were really up to handling hardware accelleration. Or at least a significant part.

Heck Nvidia and ATI cards were brand new at the time. Voodo cards were the rage...

And why not now? Because SOE has promised that they would keep the game runable on these older machines, with older video cards. Therefore they cannot add hardware acceleration in any significant amount.

Of course this may change in the graphics upgrade I keep hearing a bit about here and there... (there was a badly done blurb about it in a video game magazine where they showed a before and after pic... the after pic was suposed to be of Kelithin and looked like it would be kelithin, the before was SUPPOSED to also be Kelithin, but was the Aviak Tree city in South Ro ) but that is yet to be determined.

One thing to remeber is that EQ is OLD!!

[Dunthor Warsmith]

Despite the fact that the EQ graphics engine has been rehauled (to different degrees) numerous times, it still feels like their are artifacts from the original graphics engine - the Quake (1) graphics engine.

Maybe someone should throw glquake.exe their way

[KyrosKrane]

Earlier this month, or perhaps late last month (I forget), Absor confirmed that they're developing a new graphics engine code-named Tunare. If you'll remember, the new graphics engine that was released with Luclin was code-named Umbra.

The screen shots Ngreth refers to were a total flop. They were supposed to be before and after pictures of Kelethin, using the old and new engines, respectively. However, the "old" Kelethin was in fact a screen shot of the Aviak towers in South Karana -- you could even see an aviak strutting past. The "new" Kelethin looked to be an incomplete model; there was nothing where the sky or treetops should have been, which made it look like Kelethin was build on upside-down trees. Having said that, it was immediately clear from the screenshot that the graphics quality was a huge step up from current EQ graphics.

There have been conflicting reports about when the Tunare engine will be released, if at all. At the last Fan Faire (when they first mentioned it), the developers weren't even sure if they would release it at all, but they said then that if they did, it would be as part of a new expansion before year's end. Obviously, no new expansions have come out yet. In addition, later statements would suggest that they plan to release the new engine to everyone, not as part of an expansion.

Claims about what the engine can do also vary wildly. The developers have claimed that it will allow you to run EQ on the same PC you had when EQ first came out, and still get better performance than you would if you ran today's EQ on today's top-of-the-line PC. Other, equally astonishing, claims have made the rumor circuit, though I have no clue which should be given credence.

My personal guess is that they're just going to rewrite the entire client. The bulk of the EQ files on your computer are simply graphics files that describe zones, people, and objects; the code that handles the game mechanics is far smaller. I wouldn't be surprised if they just extract the game mechanics code, write a brand-new client from the ground up, and add the mechanics code back to that.

In short, I would certainly hope we see this soon. If even half the promises made and rumors spread regarding this new engine are true, it should make for a great upgrade to EQ.

[Minta Rose]

Thank you, Kyros, for the idiotproofed explanations!

[Itek]

*sigh*

Hardware acceleration can be turned ON. (eqclient.ini UseTnL=True) In fact for the 3d targeting ring to be visible it has to be ON. (I had it off, no ring, turned it on, ring.)

No, it doesn't use ALL the available acceleration. (generally because it's trying to stay backwards compatable, as well as trying to keep pace with hardware would be a daunting task... remember SOE thinks coding is hard)

The game INTENTIONALLY tries to hog resources. For a reason. Well... two.

First because the game doesn't have a way to anticipate.

Second because they'd like to keep the game from being hacked.

explainations

One, how do you know that since you are using 40 percent of the CPU to run the game THIS second that in the NEXT second 40 people won't suddenly appear with nifty armors, particle weapons, singing 6 bard songs and all clicking their JBoots as they run by? You don't. So as a pre-emptive strike against sudden lag (sudden massive lag = crash) the client attempts to make sure it will have as much resources at it's fingertips as possible.

This is generally -a-good-thing-(tm)

Two, if the computer has other resource hogging programs (compilers / de-compilers / hex editors / zone viewers / network sniffers / etc etc) running .... that might be a problem.

Be honest. Does it matter to you if 1 percent of the CPU time is "Idle" or 40 percent while you are running EQ? No. All computers wait at the same speed. Unless you are running something in the background that "can't wait" for you to alt+tab to it (there-by giving it de facto priority) the resource hogging isn't a problem. It only matters if you want something "running in the background" to continue working at full speed.

Realisticly there's not a legitimate reason to have multi-threaded applications running while playing EQ. Window out to check a quest, read EQTraders, check an email... sure. But if you're compiling a program why did you log into EQ while you waited?

Legitimate User: I need to upgrade my computer.
Itek: Huh? Why?
LUser: My old P3-833 isn't fast enough.
Itek: What are you talking about? Don't you mainly do word processing and work in the database?
LUser: Yep. But it's slow.
Itek: Your P3-833 is slow word processing?
LUser: No, the database is slow sometimes.
Itek: Oh, but your computer only ACCESSES the database. The database is slow, at peak times, because of the number of people in it, and the other network traffic. If you want faster database response we need a faster database server and a larger network, or a divided network.
LUser: Here's my bosses permission for a P4 1.9Ghz machine. When can I have it.
**Itek wanders away to gibber quietly about Legitimate Users**

ALL computers WAIT at the SAME speed.

It doesn't matter if EQ takes 40 percent, 45 percent, 90 percent or 99 percent of system resources. If you aren't doing something with the unused resources... what does it matter? If you ARE planning to do something with the unused resources... why are you logged into EQ??

[Aandaie]

Actually, what you mistake for being bad is

GOOD.

Its GOOD that a game like EQ uses all of the CPU and memory, when it can (outside of there being a memory leak.. or something.)

That means that it was programmed to run as well as it can based upon how good your system is. This is MUCH better than if you had a 3 ghz 800 FSB P4, with 1 gig of DDRAM and a GEforce FX 900 256 meg with EQ only using 200 meg of ram and 50 percent of the cpu.

As Itek says, you are not doing other stuff anyway.
You are not going to be rendering something complex while also playing EQ.

I much prefer EQ making use of EVERYTHING, to run as well as possible, than when I play FFXI and it only uses a max of 332 meg of my 768 meg of ram, while not running that great. I'd rather have it be made more to scale and use all of my ram that it could. For example, Anarchy Online runs better in some ways and uses as much as 699 meg of my 768 meg of ram (talking about system wide, at worst when AO was running, not what AO it self might be using.) I had luclin models off in EQ, so the only time while I approached that in EQ was when having two characters in a duo on the same computer with EQwindows.

Umbra was the code name for Luclin? Did not know that. Umbra was actually the graphics trick they used in the rendering, its called Umbra. Umbra is the technique of having anything that the user/character is not currently seeing (Such as a building obscured by the building in front of you), not be rendered. This makes your computer do much less work and improves performance a lot.

The problem in EQ was that newer zones, like Shar Vahl and Shadow haven, were made specifically FOR umbra and worked with it quite well so that if you had clipping down and such you'd be rendering far less of the area and have a higher FPS. Meanwhile. a zone like freeport still does not use Umbra at all. This is odd (you can tell since you see just as far with clipping adjusted, for one thing)

Before I left EQ, they finally had clipping and umbra working well in combination in the Plane of Knowledge. I noticed the difference right away the day they patched it, since my peformance there trippled and I could not see as far (I usually have clipping set for around 5 or 10 percent.)

Something else bad I notice in FFXI, is that the clipping doesn't do much. I go into cities and turn it way down but its only affecting geometry REALLY far away or the scenery way in the background which you actually can't get to (dummy "edge of zone" scenery.) So I still don't do that well in the complex cities. If they had clipping do more than that, I'd be playing super well.
I believe this is because its a hardly altered PS2 port and on every PS2 it would play the same, so it wasn't made to scale as well.

If the Tunare engine, as krys mentions, TRUELY is made for today than EQ should run a TON better. As with Anarchy online, the more you can adjust the better. With AO clipping works well with umbra like tech in EVERY zone, as well as the game allowing you to even turn off other players being rendered near you (great in crowded cities.)

Luclin really bloated the heck out of EQ, while not really upgrading how the old world worked with the newer stuff like umbra/clipping at all.

As far as EQW, it is MORE of an expanding resource hog. Once again, THIS IS PREFERRED. With two EQW's running, in a character duo on the same computer, I'd use up to 690 meg of system ram while only using AT MOST 40 meg of the swapfile. THIS IS GOOD. Meanwhile, with two regular SOE EQs in a duo, I'd only use around 350 meg of system ram and a WHOPPING 450 meg of the virtual swap. Real system ram is ALWAYS better than swap. So EQW is actually better at making use of what you have. A program like Cacheman (www.outertech.com), which is still free for personal use, is good for seeing how much ram/swap you used at worst after your gaming session (as well as generally tweaking some things better for gaming.)

The reason why EQ is so laggy in the bazaar is that its sticking a billion models on the screen and clipping/umbra seem to do little for player models. What is needed, more than anything else, in that zone in particular is a CHARACTER DISPLAY slider like they have in AO. If you had a character display slider that went from a few feet in front of your character, to way out there, it would not make a difference if there were 5 vendors in the bazaar or 500 since you'd only be showing the 10 or so near you (if that was all you wanted to see.)

[Dunthor Warsmith]

Originally posted by Itek
It doesn't matter if EQ takes 40 percent, 45 percent, 90 percent or 99 percent of system resources. If you aren't doing something with the unused resources... what does it matter? If you ARE planning to do something with the unused resources... why are you logged into EQ??

The Bazaar is what changed that. For people with one computer, one comp capable of running EQ, or just not willing to go through the hassle of syncing bazaar price lists, running EQ minimized in /trader while doing other work is very tempting. That's when most of my complaint with EQ being a resource hog comes from - it severly limits what you can do with EQ sitting in the background, when it doesn't seem like it needs to limit it to such a degree.

There's also the annoyance of the vastly increased load times to "just quickly check that website" while EQ is running, but I've given up on that and just use a second comp to browse the web while in EQ - but I imagine that's not an option for plenty of people.

[KyrosKrane]

Dunthor hit the nail on the head.

The moment you concede that there is ANY reason to run another program in the background, EQ no longer has a legitimate claim on all the system resources it can grab.

Maybe I want to look up a quest or tradeskill combine without waiting a full minute just for my browser to come up from minimized mode. Maybe I want to check for email without waiting thirty seconds between pressing the "Send/Receive" button and actually having the client start looking for my mail.

Maybe I want to run two instances of EQ on my computer -- whether it's for a bazaar mule or to two-box a cleric for my raid.

Maybe I want to run a macro or hacking program and cheat at EQ.

The point is, EQ can't differentiate between these actions. (Yes, there are legitimate concerns about using a hacking program, but until EQ can identify such programs, it has to assume that any program the user wants to run is permitted.) The moment I Alt-Tab out, I'm telling EQ that I want to go do "something else." At that point, a well-written program should yield any system resources it doesn't actually need so the user can get his or her work done.

As to how much resources EQ needs ... I'll be the first to admit that EQ can have spikes in how much CPU time or memory it needs, but there are a few things to note.

First, EQ doesn't need 99% CPU access all the time. Going back to my previous (simplified) explanation, a program can yield priority back to the operating system as soon as it's done. It doesn't have to wait until the end of the X milliseconds it's been assigned. So, if a program was assigned, say, five milliseconds, and it gets its processing done in two, it can yield priority at once. It doesn't have to sit there and twiddle its thumbs for the remaining three. This is what EQ does -- it never yields priority until Windows grabs it away by force. Other programs will yield priority whenever they're done with their bit of processing, or in any case after a certain interval of time has passed. The proof of this is the little hacking program I mentioned earlier. When it's run, you notice no change at all in how EQ actually plays, but if you monitor CPU usage, it drops WAY down.

Second, just because a program yields priority doesn't mean it can't get that priority back when needed. When the situation demands, EQ can use up all available CPU time. If 50 people all casting spells suddenly appear in front of you, EQ can choose to not yield priority back to the operating system until Windows grabs it back. When those 50 people zone out, though, there's no call for EQ to keep those resources it grabbed to display the 50 people. At this point, EQ should yield back the CPU time. If they zone back in again, EQ can again go back to high-processing mode.

Third, I've focused almost exclusively on CPU time in my discussions. Memory is a whole other ball game. In general, the choice of whether to use actual system memory or paged memory on the hard disk isn't up to the individual program; it's up to Windows itself. As far as programs are concerned, they just see a massive memory pool of, say, 2GB. Whether that memory consists of 256MB actual RAM and 1792MB on the hard disk or 1GB each of RAM and HDD space is irrelevant (and largely invisible) to the individual programs -- that's the whole point of virtual memory.

There are programs that can optimize how Windows uses memory (or at least, they claim to do that -- results have been mixed at best), but to my knowledge, neither EQ nor EQWindows implements that. How much actual memory and how much paged memory each uses is a coincidence. (I could be wrong on that last point, but I've seen no evidence that EQW does anything at all other than intercept the commands to draw on the screen and put them into a window.)

In short, there were once valid reasons for EQ to be a resource hog (though again, I don't agree with those reasons). Those reasons largely don't apply any more. It would make sense for EQ to be updated to modern standards, and again, I'm reasonably sure that's what they'll be doing over the next few months.

Oh, and Minta, you're more than welcome! Your posts always make me smile, so it's nice to be able to repay that in some small way.

[dahna]

Oh well, if I knew what was the Umbra option one year ago...

As a weak computer owner, I tried to optimize to the maximum the eq content over the years (low graphics, filters, no sound), and I automatically switched many options off in the eqini even if I didn't know their use.
But as SOE is giving us the opportunity to customize eq more and more (hotbutton ini files, ui, character ini, clustering main files to .xmi, .eff, .txt), they don't give us a lot of informations about them (last to date is the TargetIndicator.ini and the AttackIndicator.tga) and they leave us to the old try&error. It would be nice to at least update the doc file.

Also for the memory use, I noticed that reloading UI (same ui or another) consume more ressources that are not set free by eq unless you exit it : so loading ui 5 or 6 times for me (have 512Mo and logging to eq use ~300) result in a window memory warning message (=return to desktop from full window mode).

[Sanna]

hm, lots of info, late-ish at night. technically, i should know better than to respond now ^_^ if i think i should sleep.


Yes, and especialy in trader, EQ should be able to run with other programs. I should be able to watch .avi's and listen to music. Yeah, i understand the thing about 'well, maybe lag will happen soon and EQ needs to be prepared,' but... I'm in trader, staring at the ground. is there really not a better way to do it?

I don't know, even if i got a 100pct correct in all aspects answer to that, 99pct chance I won't understand it, heh. Anywho, thanks for the answers.




Edit.
THAT was the other thing i forgot.
I was trying to edit around the playerwindow xml file in a skin, and i had a hotkey for /loadskin whateverfile 1. I would make little changes and reload the skin (I'm still SUCH a noob at ui editing and yes I've tried to get help at eqgui.com). Eventually, EQ just SHUT DOWN. Boom. Desktop. No message, no nothing. It just decided it was not going to run anymore. I was greatly amused ^_^!!!!!!

-- Sanna
and Merry Christmas. ^_^
PS:

[Malathos]

ALL computers WAIT at the SAME speed.

I just finished a contract where someone on the team complained that their PC was way too slow - and that some piece of software called "System Idle" was using up 99% of their machines CPU time - oh, and this person was a software developer