Following the release of the Dominion expansion we started to get reports from players about an increase in "fleet lag." Before we go any further, I would like to assure everyone that a taskforce of developers has been working on this continuously, but tracking down the source has proven a very difficult task.

So, what is the problem?

"Lag" is an ambiguous term and can refer to many things. The graphics in the client might be slowing it down, the network might be congested or a host of other issues may be coming into play. We have spent a long time trying to reproduce the issue on the test servers as well as monitoring fights as they have been unfolding on Tranquility, where we have received help from the players in the form of testing and bug reports. You might see names like "CCP Atlas", "CCP GingerDude" and "CCP Warlock" poking about in Local. Don't be alarmed, we are simply monitoring the node, looking for glitches. There are senior software engineers spending their evenings and weekends on Tranquility monitoring and measuring since we have not been able to see issues when testing because the sheer number of people required to trigger the problems and the specific 'usage pattern' of the game systems in question are outside what can currently be tested in the lab (although this is something that is being worked on).

What we are looking at now specifically is the fact that the performance of the sol nodes (the computers in the clusters handling the solar systems) appears to be worse than before. Even though CPU across the cluster has not increased with the Dominion expansion, each reinforced node is no longer able to handle the same number of people fighting as gracefully as it did before.

Additionally, we have the issue of the grid not loading when jumping into a system. Being stuck on the source side and not getting through the gate is one thing since you're not in the battle yet, but if you actually get through the gate and arrive on the other side without managing to load the grid you could be at a disadvantage and possibly vulnerable because of your client's inability to act. This can be detrimental to the fleet jumping into the system.

What is happening is that the client is requesting information about the simulation from the server but it's not receiving it in a timely manner. There may be many minutes before the information is finally received as it is getting stuck in the lower levels of the sol node.

Anatomy of "grid not loading"

When the specific scenario of "grid not loading" occurs, its fingerprint on your client is quite specific (even though sadly we cannot see the effects in the server logs).

Refer to the following handy diagram to determine whether you have encountered the problem:

This is what it would typically look like if you're jumping into a system which exhibits the issue. However, if you are logging into such a system and encounter this issue you will only have a black screen and a "loading" bar.

If we look at the image a bit you will see that the local channel, the right click menu and the system information are all updated to the new system while the nebula is that of the old system. Your overview will also be empty and no brackets in space. If you open up the monitor (control-shift-alt-m) it will have (at least) one outstanding call. This is the request for the simulation instance.

When you see this, the worst thing you can do is to click any buttons or type in local. It will only exacerbate the problem. Relogging will also not help and might even make the situation worse. The best you can do is wait it out (see the "Fleet fight survival guide" below).

Light at the end of the tunnel

We were finally able to reproduce this with a special fleet test on our Singularity test server on Jan 27th where over 400 people participated in helping us out. That was really fantastic to see and I would like to thank everyone who showed up.

On Jan 28th, we had a 1600 person fleet fight on Tranquility which our team monitored closely, keeping the node alive using methods that make our system admins faint. This was one of the biggest, if not the biggest, fleet fight ever in EVE (at least where the node survived the ordeal). This event allowed us to identify what was causing some of these glitches and deploy fixes live.

We have added a stop-gap measure in place which keeps nodes from dying during these high stress scenarios and we have some more permanent solutions in the pipes that will be deployed over the next few weeks.

How can I help?

There is a mass test scheduled for the Singularity test server on Thursday Feb 4th at 16:00 Eve time. Those of you who wish to help out, please come and provide us with client logs when you encounter the "grid not loading" problem or other issues. We will specifically be testing jumping in several solar systems trying out a few of the software solutions that have been implemented.

This requires running Logserver.exe from the client installation folder before you start the client. Your logs are extremely important, and each bug report you submit with this information will go a long way toward bringing us closer to a resolution or fix for this issue.

There is more information on this fleet test in this thread.

Additionally, if you are planning to attack a system we also highly recommend that you notify us at least 24 hours beforehand through the Fleet Fight Notification Form so that the system can be placed on a reinforced node. This will also give us the chance to mobilize people to be in the solar system with measuring devices firmly attached to our Polaris frigates.

Fleet Fight Lag Survival Guide

I'd like to talk a little bit about the best tactics we've found to use during periods of extreme server-side lag and methods to make the experience a little more bearable. Note that I am talking about server-side lag here, not low FPS or bad network conditions (for example, if you find yourself in a fleet fight on your 9600bps cell phone modem up on a glacier somewhere). The lag that this refers to is when there are hundreds of players fighting in a solar system and everyone is experiencing delays, where guns could take minutes to fire and there's some gnarly rubberbanding. If you've encountered this type of lag you typically know the symptoms. It's under these circumstances that the "grid not loading" issue manifests itself for example.

Do not hammer that button!

One of the most important things to keep in mind when you are in a system with heavy lag is that you should not hammer any buttons and you should try to execute as few commands as possible. If you are waiting for the grid to load the worst thing you can do is click buttons on the interface or chat in local. These actions will have a negative impact on your client's ability to recover (more to the point, on your session on the server). If you have an outstanding call you should wait for it to complete, and it will complete eventually. It might take minutes but your gun will fire and clicking the button again will not help. Do not browse the market or look up contracts while waiting for the system to load. I know it's hard but you have to leave the client alone while it waits for the calls to complete or you will add to the problem and can even cause the session to get stuck.

Use the Monitor

The network tab in the ingame monitor debugging tool (control-shift-alt-m) can help you understand whether your calls are completing in a timely manner. If there is a number in the "Outstanding" row that means that the server is busy executing your command. Wait for it to complete before executing another one!

Calls do time out

What I said about calls completing eventually is true... from a certain point of view. The call will be handled eventually by the server but it's possible that the client has given up by then. Calls typically time out in 8 minutes and this is true of the request for the simulation instance (grid). If you have waited for 8 minutes for the grid to load you will not recover. If you have the monitor open and see the Outstanding number drop to 0 and your grid doesn"t load then you have no choice but to relog, and even that might not fix things if the server is still overloaded.

Note that if you get podded while in this state then the client should recover when your session is moved into your clone station.

Final words

We have come a long way in understanding the issues that have plagued large fights after Dominion and have gotten important insight into fleet fight lag as a whole. The problems we are experiencing post-Dominion are being worked on and they will be fixed soon, adding more space carnage to your battles.

I would like to ask that we keep the discussion thread attached to this blog productive and try to keep flaming to a minimum.

Thank you,

Jon Bjarnason
Technical Director
EVE Online, CCP Games

A Word from Customer Support

Support's intervention in large fleet engagements in even a remotely fair manner is very problematic due to the complexity of the situation.

Under circumstances where lag is involved or elusive bugs that are difficult to establish as the deciding factor, CCP cannot justify passing judgment on who should have won or who should have lost. While a problem such as the one in focus here is known to affect some players in certain situations, the evidence available in server-side logs, when available at all, can be haphazard and arbitrary. As such, intervention on CCP's part would risk being arbitrary as well.

What this means is that CCP will not be granting reimbursement for fleet fight losses.

Please understand that fleet fight reimbursement has always been very controversial and few issues have been discussed and argued in more detail within CCP. The conclusion has, however, always been to leave fleet battles alone rather than reimbursing them as a whole. We apologize for any inconvenience this may cause and we appreciate your patience as we work diligently towards resolving this issue.

The amorous folks at Battleclinic are getting into the swing of Valentine's Day early with a great contest. Let them know why you love EVE Online by using candy hearts. Head on over to Battleclinic and show them your sweet tooth for a chance win EVE Store coupons or two weeks of game time.

Sahtogas, The Bleak Lands

The Amarr system of Sahtogas fell to Minmatar occupancy yesterday at 14:55.

Sahtogas controls the only access point to the Amarr system of Haras, occupancy of which is currently fiercely contested between the Minmatar Tribal Liberation Force and the Amarr 24th Imperial Crusade.

The solar system is reported to house considerable mineral wealth, with surveys suggesting the presence of Promethium and several other valuable metals in the local moons.

While 24th Imperial Crusade forces had resisted Minmatar invasion of the system by protecting defensive installations throughout Tandoiras constellation, the final siege of the 24th Imperial Crusade bunker in the system slipped by with no active resistance and its fall passed unannounced on capsuleer GalNet channels.

31-MLU, Syndicate

Near 06:00 this morning, over 100 corporations were expelled from the GoonSwarm [OHGOD] alliance. Issachiar, pilot of a Hel-class supercarrier for GoonFleet, was caught by the dissolution of his alliance while within the shields of a starbase belonging to Battlestars corporation. The Hel was expelled from the shields where waiting forces from Cry Havoc [C H] fell upon the vessel.

When corporations began dropping out of GoonSwarm, Cry Havoc began receiving automated surrender messages. Mduncan0341 of C H described how those messages became a supercapital kill for him and his wingmates.

"We were on a roam, when we noticed Goonswarm alliance disbanding - our mailboxes were being flooded with surrender notices, since we have an active wardec against them. We ran back to Orvolle to ship into [remote repair battleships] since we knew they had a supercap staging [tower] in 31-MLU and we knew caps would bounce as they [arrived]."

While GoonFleet members VCtab and Apples Appleby claimed the Hel's death was a desperate, deliberate and suicidal act from a pilot devasted by the loss of his alliance, eyewitnesses in Cry Havoc stated that the supercarrier was repelled by the Battlestars' tower's shield and recoiled a substantial distance, at least in terms of supercapital speeds - an accidental casualty of GoonSwarm's shattering.

While the Hel at first managed to evade the C H fleet as they jumpbridged in, the ship was uncloaked at a nearby moon by a scouting Dramiel. It was tackled and the Cry Havoc raiding fleet quickly called in their standby capitals, including a pair of titans and a supercarrier of their own.

The former GoonSwarm corporations sent multiple waves of ships in an attempt to break the Hel out from the interdiction the C H fleet had put upon it, using Interdictor warp disruption "bubbles" and the infinite strength warp scrambling of Heavy Interdictors. The now-combined fleet from Cry Havoc and allies Ev0ke, however, was able to destroy the Hel in approximately 6 minutes and turn their attention to evacuating their own supercapitals before a heavy fleet from GoonSwarm's so-called "CapSwarm" could respond. In the end, C H were able to depart the field with very light losses.

The expulsion of GoonSwarm's corporations has caused the collapse of the alliance's remaining sovereignty in Delve and Querious. (Other systems were lost on 26.01.112 when GoonFleet corporation was unable to make necessary upkeep payments.)

Multiple sources within the former GoonSwarm corporations have claimed that the corporations were expelled by karttoon, the former alliance leader and CEO of GoonFleet. Karttoon was allegedly one of three people who could have transferred funds to the correct division of the right corporate wallet to make the sovereignty payments, but he was absent at the time, as were the two other officers with wallet access.

Another source in GoonFleet speculated that mass demands for karttoon's resignation in the wake of the lapsed upkeep payments in Delve may have motivated his alleged actions.

In addition to expelling the GoonSwarm corporations from the alliance shell, karttoon is alleged to have emptied the corporate and alliance wallet divisions that he could access, taking sums reported varyingly as 300 - 500 billion ISK in liquid funds and a possible trillion ISK in assets.

Apples Appleton claimed karttoon then went on to detonate several ships from the alliance dreadnought cache in order to claim the insurance, but other sources have queried that descriptions of the destroyed ships showed that their names spelled out an insulting message to karttoon.

In the wake of GoonSwarm's dissolution into its constituent corporations, there is much speculation on what they will do next. A reported statement by Darius JOHNSON, former GoonSwarm leader and likely leadership candidate for any replacement, says:

"As many of you may have noticed we no longer have an alliance. The very gentleman who is CEO of fleet is the likely culprit so we'll be dropping Goonfleet as well. Goonfleet is dead. Goonswarm is dead. We will now bask in the glory of Goonwaffe."

GalNet References :

Cry Havoc Battle Record

Lapsed Upkeep Payments Remap Null-Sec Sovereignty

Hulm, Heimatar

In the early hours of this morning, 03.02.112, GoonSwarm alliance disbanded.

All but two corporations from the GoonSwarm alliance are currently flying under no alliance ticker. The Goony Hand Social Club banking corporation remains in the shell of the alliance after trouble accessing it in the absence of GoonFleet CEO Karttoon led to the loss of sovereignty over key GoonFleet systems in Delve.

Reasons for the move are not yet known; given the recent decision to abandon the alliance's conquered space in Delve and Querious and return to their roots in Syndicate, coupled with the recent return of former alliance leader Darius JOHNSON, a restructure of the alliance seems possible, but no statement has been made at this time.

More news on the dissolution of GoonSwarm will follow as it comes in.