Author Topic: Launch Day Numbers  (Read 16515 times)

Codewalker

  • Hero of the City
  • Titan Network Admin
  • Elite Boss
  • *****
  • Posts: 2,740
  • Moar Dots!
Launch Day Numbers
« on: July 08, 2015, 03:45:05 PM »
Since a few people asked, here's a few statistics gathered last night.

The maximum number of concurrent users peaked at around 200. At that time, the XMPP server's CPU usage was about 20-25% (around 1 full core on a 4 core system). Outgoing bandwidth was around 8Mbps at that time, and incoming averaging 2Mbps. Peak traffic was 14Mbps.

At one point we had around 120 players in Atlas. The one hiccup we had was when about 100 of those who were following the livestream all tried to mapmove to Pocket D at once. That caused something to go wrong on the server and it got stuck in a loop using 99% CPU and was unresponsive - chat traffic wasn't being passed. I restarted the service, which kicked everybody out momentarily, and re-established the default 30 player cap for Atlas. Once everybody got back in we had 3 Pocket D instances and 3 Atlas Park Instances.

After things had settled down a bit, I changed Atlas and Pocket D back to being uncapped. So far the issue has not repeated itself. When I get some spare time I'll dig into the server logs to see what it got stuck on.

Since the launch, 788 unique accounts have logged into the Titan XMPP server. No doubt most of those were using Paragon Chat.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Launch Day Numbers
« Reply #1 on: July 08, 2015, 06:08:44 PM »
The maximum number of concurrent users peaked at around 200. At that time, the XMPP server's CPU usage was about 20-25% (around 1 full core on a 4 core system). Outgoing bandwidth was around 8Mbps at that time, and incoming averaging 2Mbps. Peak traffic was 14Mbps.

So about 50kb/sec per connection (40 up, 10 down).  So the 25kb/sec - 100kb/sec per user was actually a pretty good estimate.  Yay, I'm not an idiot.


Quote
At one point we had around 120 players in Atlas. The one hiccup we had was when about 100 of those who were following the livestream all tried to mapmove to Pocket D at once. That caused something to go wrong on the server and it got stuck in a loop using 99% CPU and was unresponsive - chat traffic wasn't being passed. I restarted the service, which kicked everybody out momentarily, and re-established the default 30 player cap for Atlas. Once everybody got back in we had 3 Pocket D instances and 3 Atlas Park Instances.

I'm honestly not sure what would happen if 100 players in the same zone simultaneously tried to execute a mapmove command to the same destination on the live servers.  However, it sounds possible that with everyone standing still and appearing in the same place in visual range, there was a sudden exponential burst of metachannel traffic that broke the room.

Still, pretty good for day one.

epic

  • Underling
  • *
  • Posts: 6
Re: Launch Day Numbers
« Reply #2 on: July 08, 2015, 06:23:40 PM »
Since the launch, 788 unique accounts have logged into the Titan XMPP server. No doubt most of those were using Paragon Chat.

That means 788 people in the community still care. Those are nice numbers.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Launch Day Numbers
« Reply #3 on: July 08, 2015, 06:25:53 PM »
That means 788 people in the community still care. Those are nice numbers.

Some might have been chat clients, but conversely there are still a lot of people either having difficulty setting up Paragon Chat, just plain downloading Paragon Chat or the I24 client, or having other technical difficulties that make it impossible for them to log in yet.  I would not be surprised to see that number double in a week or so.

Zekiran

  • Lieutenant
  • ***
  • Posts: 98
Re: Launch Day Numbers
« Reply #4 on: July 08, 2015, 07:04:46 PM »
It was positively amazing. Having been part of the hours before the release, seeing just the few of us and then hearing the livestream... It was full of amazing.

I was wondering what was going on - watching the stream and seeing everyone separately hit D and then Ouro, and all having the same response, "um, there's no one here and my chat stopped working" was pretty telling. Obviously still some bugs to be worked out there. But on the whole, relaunching when invisible to people seemed to work *mainly* (I'm guessing that as time went on and more people got on, that became the issue rather than just being slightly out of synch - since chat worked, but visibly the characters weren't there for some, were for others).

780+ people *in the first night* seems like a marvelous indicator of how this will go.

Now if only the folks having issues (the same issues, usually) will come HERE instead of randomly asking in the facebook groups for help...

Kheprera

  • Elite Boss
  • *****
  • Posts: 278
Re: Launch Day Numbers
« Reply #5 on: July 08, 2015, 07:26:00 PM »
I could not yet log in. My computer is still in Wisconsin.

I'm in Texas.

Thus me logging in is difficult.

Also none of my SG has yet logged in (though one is a Cape DJ and imagine he'll be there soonish). I ramble.

Some people on Facebook were getting errors. I tried to send them here to report errors for tracking.

Yay for success!

Blackmoor

  • Underling
  • *
  • Posts: 9
Re: Launch Day Numbers
« Reply #6 on: July 08, 2015, 07:27:52 PM »
That means 788 people in the community still care. Those are nice numbers.

And that's only considering the people that still active pursue CoX news. There's an even larger number of people that still care, but they aren't keeping their ears to the ground like we are.  :D

Noyjitat

  • Elite Boss
  • *****
  • Posts: 717
  • Guess who cares?
Re: Launch Day Numbers
« Reply #7 on: July 08, 2015, 08:35:46 PM »
Since a few people asked, here's a few statistics gathered last night.

The maximum number of concurrent users peaked at around 200. At that time, the XMPP server's CPU usage was about 20-25% (around 1 full core on a 4 core system). Outgoing bandwidth was around 8Mbps at that time, and incoming averaging 2Mbps. Peak traffic was 14Mbps.

At one point we had around 120 players in Atlas. The one hiccup we had was when about 100 of those who were following the livestream all tried to mapmove to Pocket D at once. That caused something to go wrong on the server and it got stuck in a loop using 99% CPU and was unresponsive - chat traffic wasn't being passed. I restarted the service, which kicked everybody out momentarily, and re-established the default 30 player cap for Atlas. Once everybody got back in we had 3 Pocket D instances and 3 Atlas Park Instances.

After things had settled down a bit, I changed Atlas and Pocket D back to being uncapped. So far the issue has not repeated itself. When I get some spare time I'll dig into the server logs to see what it got stuck on.

Since the launch, 788 unique accounts have logged into the Titan XMPP server. No doubt most of those were using Paragon Chat.

So it would seem to be safe you need a system in place that only allows so many people to change zones at once. some sort of queue that occurs during a loading screen. 30 is the default cap per zone or is that the cap for atlas only? Because I remember having 50 in rikti war zone and the hive for full league and two people outside of the league.

If you think we'll need better hardware start a donation page. I imagine you'll meet the required goal in record time haha.

Stealth Dart

  • Boss
  • ****
  • Posts: 110
  • Martial Artists kick butt with style
Re: Launch Day Numbers
« Reply #8 on: July 08, 2015, 08:48:31 PM »
It was really great seeing all of you who made it in.   :D

To those having difficulty I hope you persevere until you are successful.  It was a real tear jerker moment  :'(  logging in to Atlas Park and seeing members of our community livening up the place.  Looking forward even more to what is yet to come.  Thanks Codewalker and all who helped to make this a reality.  I really, really miss my home.  So great to have our community back!   ;D
I am a dancer, a leaf in the wind...a leaf that can kick your Butt!

Golden Aurora

  • Boss
  • ****
  • Posts: 108
Re: Launch Day Numbers
« Reply #9 on: July 08, 2015, 09:02:03 PM »
I had screenshots the night/moment the servers went down, and again the moment CW made them come up.
It was a night I will remember for a long, long time and hopefully the beginning of the return of something we've all lusted after.

It's good to be home.

Remaugen

  • Elite Boss
  • *****
  • Posts: 457
  • Android Clan of One
Re: Launch Day Numbers
« Reply #10 on: July 08, 2015, 09:16:39 PM »
Myself and one other person ended up in Pocket D4.
We're almost there!  ;D

The RNG hates me.

Brightfires

  • Elite Boss
  • *****
  • Posts: 269
  • ... is *NOT* Groot
Re: Launch Day Numbers
« Reply #11 on: July 08, 2015, 09:22:04 PM »
It was great fun seeing everyone last night... Even if they *couldn't* see me.  ;D

(Apparently both Palrah and Kestrel were invisible to the majority of the other people in Atlas and the D. 'Something to do with their costumes, it seems. My non-angel-winged Nemissary showed up just fine.)
Taker of screenshots. Player of bird-things.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Launch Day Numbers
« Reply #12 on: July 08, 2015, 09:23:33 PM »
So it would seem to be safe you need a system in place that only allows so many people to change zones at once. some sort of queue that occurs during a loading screen. 30 is the default cap per zone or is that the cap for atlas only? Because I remember having 50 in rikti war zone and the hive for full league and two people outside of the league.

That would probably require hacking the Openfire server, which I think Codewalker would rather chew through his own leg than do.

Its probably not the explicit switching of zones that's the problem, but what happens next that is the problem.  Everyone (i.e. every game client) now has to figure out where everyone else is, and what they are doing, simultaneously.  A more sophisticated discovery protocol might address that in theory without having to tamper with the XMPP server, but that might be a sufficiently complex TODO to make it a lesser priority.

Its also possible there's a subtle design flaw in the way presence messages are processed in high density situations that could be resolved algorithmicly.  I'm certain Codewalker will find a solution in due time.

Surelle

  • Guest
Re: Launch Day Numbers
« Reply #13 on: July 08, 2015, 09:36:33 PM »
Some might have been chat clients, but conversely there are still a lot of people either having difficulty setting up Paragon Chat, just plain downloading Paragon Chat or the I24 client, or having other technical difficulties that make it impossible for them to log in yet.  I would not be surprised to see that number double in a week or so.

True, me among them.  Glad to hear so many were able to take that maiden voyage though!  Cool beans!  People still care! 

Not complaining here by a long shot, just waiting until sometime when the dropbox bandwidth isn't drained dry.  Again, not complaining.  Even if I can't download for weeks, I'm still happy we even have this option available.  Months and years fly by like nothing to me.  I blink and they're gone-- a side-effect of getting old methinks.   :P

Graphite

  • Lieutenant
  • ***
  • Posts: 89
Re: Launch Day Numbers
« Reply #14 on: July 08, 2015, 09:49:52 PM »
We are the Hard Core Super Early Adopters.  In business speak, we're the crazies who camp outside for the Newest I-Phone.  I'm sure those numbers will climb, especially as new hits the press circuit and word of mouth starts to spread.  All and all I think it was a fantastic release.

IMHO, someone should at some point add a link in the Paragon Chat Forums that's easy to find for Tequila with out having to dig for it in another sub-folder like I had too.  I'm really impressed with how easy everything was to set up.  However, the simpler the better so people don't get frustrated enough to potentially quit.

I mentioned this in chat but am posting it here as well so that it gets properly logged.  If you AFK it disconnects you from Paragon Chat but NOT Icon.  I came back before the old time window, noticed the chat hadn't changed and that no one else was around.  Looking at the system alerts I then saw: "You went AFK.  Unknown command AFK.  You went AFK."  Again, my assumption is that AFK causes a bump off the servers since its unknown, but hey, I'm no expert.

And Again THANK YOU THANK YOU to everyone involved with this project!  It's great having our City back!
Praise the Code!

Golden Aurora

  • Boss
  • ****
  • Posts: 108
Re: Launch Day Numbers
« Reply #15 on: July 08, 2015, 09:59:50 PM »
That would probably require hacking the Openfire server, which I think Codewalker would rather chew through his own leg than do.

Its probably not the explicit switching of zones that's the problem, but what happens next that is the problem.  Everyone (i.e. every game client) now has to figure out where everyone else is, and what they are doing, simultaneously.  A more sophisticated discovery protocol might address that in theory without having to tamper with the XMPP server, but that might be a sufficiently complex TODO to make it a lesser priority.

Its also possible there's a subtle design flaw in the way presence messages are processed in high density situations that could be resolved algorithmicly.  I'm certain Codewalker will find a solution in due time.

Actually Arcana, I can think of ways to do this.
You could have a room that sits between your source and destination room.
Call it a loading room if you will.

While within this room, the only thing you have to broadcast is your destination and timestamp of when you entered the loading room.
First come, first serve for when you leave the room and go to your destination.

While in this room you wouldn't need to flood the server asking for costume pieces, and all that other stuff that is the cause of the server issues.

Arcana

  • Sultaness of Stats
  • Elite Boss
  • *****
  • Posts: 3,672
Re: Launch Day Numbers
« Reply #16 on: July 08, 2015, 10:23:28 PM »
Actually Arcana, I can think of ways to do this.
You could have a room that sits between your source and destination room.
Call it a loading room if you will.

While within this room, the only thing you have to broadcast is your destination and timestamp of when you entered the loading room.
First come, first serve for when you leave the room and go to your destination.

While in this room you wouldn't need to flood the server asking for costume pieces, and all that other stuff that is the cause of the server issues.

That would break mapmove and force everyone to learn how to make double moves.

Risha

  • Boss
  • ****
  • Posts: 232
  • Dancing at the end of the World
Re: Launch Day Numbers
« Reply #17 on: July 08, 2015, 10:51:50 PM »
Well I downloaded, got impatient and screwed up stuff on my end and downloaded AGAIN (in a better place), so I went in late but made it.  :)  In a little today, and now I have 3 toons up.  One in Atlas, one in Pocket D, and one in the Cryptic Lounge (no NPCs in the Cryptic Lounge).

Don't know as I ever ran through Skyway before, but, man, it's not fun. :)   Somethings I've embarrassingly have forgotten, some I'm surprised I recall.  And wonderful points -- oh, I died here the first time I came into this zone.  Oh, this is where I met . . .

THANK YOU, THANK YOU, THANK YOU.
Writer of Fantasy and Fantasy Romance

laufeyjarson

  • Boss
  • ****
  • Posts: 141
Re: Launch Day Numbers
« Reply #18 on: July 08, 2015, 10:53:10 PM »
So awesome to see the client full of people again!  Thank you all so much for doing this.

It didn't feel like there were 200 people on, so we must have been spread out pretty good.  :)

Golden Aurora

  • Boss
  • ****
  • Posts: 108
Re: Launch Day Numbers
« Reply #19 on: July 09, 2015, 12:36:29 AM »
That would break mapmove and force everyone to learn how to make double moves.

I envisioned it more as an automatic process taken care of by Paragon Chat. You type in the mapmove number and it does the rest.