48

Degraded performance on GitLab.com

 5 years ago
source link: https://www.tuicool.com/articles/uYNJ3qb
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

Currently seeing a significant site slow-down and an increase in errors

Summary

  • As of Jul 1 14:01:38 UTC 2019 we are currently to investigate the slowdown on GitLab.com that is affecting all web requests that are serviced by the rails front-end. The slowdown is in redis, in the lru cluster. There were no recent application or configuration changes prior to degradation.

Current Redis topology

redis-03
redis-cache-02

E7RVBnb.png!web

Metrics controller looks very bad, but this might just be a symptom:

rqEzqq7.png!web

Increased rate of 5xx errors:

vm2UjyM.png!web

Currently seeing a significant site slow-down and an increase in errors

Summary

  • As of Jul 1 14:01:38 UTC 2019 we are currently to investigate the slowdown on GitLab.com that is affecting all web requests that are serviced by the rails front-end. The slowdown is in redis, in the lru cluster. There were no recent application or configuration changes prior to degradation.

Current Redis topology

redis-03
redis-cache-02

Screen_Shot_2019-07-01_at_10.52.32_AM.png

Metrics controller looks very bad, but this might just be a symptom:

Screen_Shot_2019-07-01_at_10.52.53_AM.png

Increased rate of 5xx errors:

Screen_Shot_2019-07-01_at_10.53.34_AM.png


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK