WARN: CloudHandler - ErizoController 0 ---- does not respond. Deleting it

Support
1

WARN: CloudHandler - ErizoController 0 in x.x.x.x does not respond. Deleting it.

Any idea how to fix this?

2

I also met this problem, using the latest code from master branch.I install licode on Ubuntu 14.04.2.

3

Does it caused by no data in mongodb?Should I add some data to it?

4

This warning can be ignored…this is caused by an old erizoController which is created by former node process…

5

Hi All - I keep getting this error too on one of my servers every few days… This happened since i upgraded to v2.

The sequence of events is always:

2017-04-12 09:53:56.477  - WARN: CloudHandler - ErizoController 0  in  x.x.x.x does not respond. Deleting it.
2017-04-12 09:53:56.538  - WARN: CloudHandler - I received a keepAlive message from an unknown erizoController
2017-04-12 09:53:56.540  - ERROR: ErizoController - message: This ErizoController does not exist in cloudHandler to avoid unexpected behavior this ErizoController will die

After this, all connections will fail with the following error:

2017-04-12 15:51:57.675 - ERROR: CloudHandler - No erizoController is available.

Any idea on what could cause this?

6

You should review why ErizoController is failing in this case every few days.

7

Hi,

I updated all my servers to v3. I have 2 in Europe and 2 in China. This issue still happens on my servers in China (one is on azure and the other on some local chinese provider). I must also say that I only saw this issue on my servers in China, never anywhere else.

Here is a snap of the logs… Last message before it happened was at 22:33:

2017-07-04 22:33:44.903 - INFO: EcCloudHandler - message: deleting erizoJS, erizoId: e50b5cf2-edc6-19fd-b304-123237253504

Then out of the blue, at 01:11, issue happened:

2017-07-05 01:11:28.958  - WARN: CloudHandler - ErizoController 0  in  xxxxx does not respond. Deleting it.
2017-07-05 01:11:28.970  - WARN: CloudHandler - I received a keepAlive message from an unknown erizoController
2017-07-05 01:11:28.981  - ERROR: ErizoController - message: This ErizoController does not exist in cloudHandler to avoid unexpected behavior this ErizoController will die
2017-07-05 01:11:28.988  - INFO: CloudHandler - [CLOUD HANDLER]: ErizoController in host  xxxxx wants to be killed.

When that happened, the MongoLog shows crazy times:

Wed Jul  5 01:11:28.813 [conn20] update nuvedb.erizoControllers query: { _id: ObjectId('5958b14b47c4f706031b1ecb') } update: { $set: { keepAlive: 0 } } idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:41 14864ms
Wed Jul  5 01:11:28.814 [conn3] update nuvedb.erizoControllers query: { _id: ObjectId('5958b14b47c4f706031b1ecb') } update: { $inc: { keepAlive: 1 } } idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:32 14864ms
Wed Jul  5 01:11:28.814 [conn2] update nuvedb.erizoControllers query: { _id: ObjectId('5958b14b47c4f706031b1ecb') } update: { $set: { keepAlive: 0 } } idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:51 13864ms
Wed Jul  5 01:11:28.814 [conn21] update nuvedb.erizoControllers query: { _id: ObjectId('5958b14b47c4f706031b1ecb') } update: { $inc: { keepAlive: 1 } } idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:23 13863ms
Wed Jul  5 01:11:28.814 [conn1] update nuvedb.erizoControllers query: { _id: ObjectId('5958b14b47c4f706031b1ecb') } update: { $set: { keepAlive: 0 } } idhack:1 nupdated:1 fastmod:1 keyUpdates:0 locks(micros) w:41 12863ms

I would greatly appreciate if anyone can help me troubleshoot this issue. Where should I look? Any other log or something i can do when it happens (since it pretty much does every few days)

Thanks

-VR

8

Happens also on my servers randomly (Ubuntu 16.04, Licode v3), really annoying :frowning:

9

Hi Mohsen,

I could notice only v2, where you are referring v3?

Please find the list of branches below:

[hemanth@localhost licode]$ git branch -a

  • master
    packages
    v1.2
    remotes/origin/HEAD -> origin/master
    remotes/origin/agents_monit
    remotes/origin/erizo_tree
    remotes/origin/erizo_tree_v2
    remotes/origin/gh-pages
    remotes/origin/legacy
    remotes/origin/logs_with_connectionid
    remotes/origin/master
    remotes/origin/packages
    remotes/origin/pipe-erizojs-logs
    remotes/origin/revert-939-refactor/erizo_client_roomjs
    remotes/origin/test
    remotes/origin/upgrade_openssl
    remotes/origin/v1.2

Thanks & Regards,
Hemanth

10

You are using code from master branch (or) some other branch?

11

Hi Hemanth,

Releases are specific commits in the master branch: https://github.com/lynckia/licode/releases

12

Thanks Mohsen.

I think it is better to tag with its version. We will suggest this.

13

also happened to me once

14

You can check the mongodb server and rabbitmq server have been started. If not ,start it by yourself.

15

Restarting EC fixes the problem temporarily. I had to handle it in my app so if EC does not respond after a while then I restart it.

16

WARN: CloudHandler - ErizoController 0 in x.x.x.x does not respond. Deleting it.

I just installed the latest Licode and I face this issue in my server environment. In my local environment which Licode is 1 month ago installed, there is no warning like this.

I try to restart the server itself but it keeps popping-up.

Does anyone solved now this issue?

17

It prevents the current functionality of Licode or everything’s working fine anyway?
Because Licode stores in mongodb each erizo controller ID and when you restart the process, it fetches the db, try to send ping to that controller and if it doesn’t repond it deletes him.
so maybe you are getting this “error” every time you restart licode but it’s fine 'cause it’s an old process controller. It should have spawned a new one with a new ID and that should handle the traffic fine.