Symptoms you may meet, what causes them, and how to look inside a service. Everything here comes from how Malla behaves; the error values are the ones the code returns.
Looking inside
These work on any running node (from iex, a remote shell or rpc):
# What does the service think it is? (a call to the service process)
{:ok, info} = Malla.Service.get_service_info(MyService)
info.running_status, info.admin_status, info.last_error, info.plugins
# Cached, cheap, :unknown if the process is not running
MyService.get_status()
MyService.get_config()
# The plugin chain, top to bottom, and the chain of one callback
MyService.service().plugin_chain
Malla.Service.get_callbacks(MyService, :my_callback)
# Which nodes run the service, and which node am I connected to?
Malla.Node.get_nodes(MyService)
Malla.Node.get_instances(MyService)
Node.list()
# Raw discovery: members of the process groups
:pg.get_members(Malla.Services2, MyService)
:pg.get_members(Malla.Services2, :all)
# Is it live / ready (what a probe would say)?
Malla.Service.is_live?(MyService)
Malla.Service.is_ready?(MyService)Malla.ServiceIdMissing ("service_id is missing")
Symptom. An exception from Malla.get_service_id!/0 when you use span, info, req, Malla.Request.request/4, Malla.metric/4, Malla.Status.public/1, Malla.authorize/3 or any Malla.local/2.
Cause. Those calls need to know which service they belong to, and take it from the process dictionary. Callbacks (defcb) and calls made through Malla.remote/4 or Malla.local/3 set it. A plain function that you call directly, a process that you spawn, an iex session or a test process do not have it.
Fix. Set it where the work starts: Malla.put_service_id(MyService), or pass the :service_id option to the call that takes it (span/3, Malla.Request.request/4, Malla.authorize/3), or call the function through Malla.local(MyService, :fun, args). As a global fallback, config :malla, default_service_id: MyService makes get_service_id!/0 use that service (and log a warning each time it does).
{:error, :malla_service_not_available}
Symptom. Malla.remote/4 or Malla.call returns it; Malla.Request turns it into the status "service_not_available". It can take about 5 seconds to come back.
Cause and fix. It means no node answered for that service. In order:
- The service is not running anywhere. Check
Malla.Node.get_nodes(MyService)on the calling node. A service is only listed while its running status is:running: not while it is:starting,:paused,:failedor:stopped. Look atMalla.Service.get_service_info/2on the node that should run it. - The nodes are not connected.
Node.list()must show the other node. See "Malla.Cluster not connecting" below. - The calling node has not discovered it yet. Discovery is pushed when a service changes status and re-checked every 10 seconds; right after connecting it takes a moment.
Malla.Node.wait_for_services/2waits for it. - The name is wrong. The service ID is the module name. With
req, analiasis not expanded, soreq UserService.get(1)asks forUserService, notMyApp.UserService(see Request Handling). - The service is paused. A paused service is not offered, and
Malla.Plugins.Requestrejects requests with this error.
The call retries before it gives up: 5 times, one second apart (sna_retries, retries_sleep_msec). Pass sna_retries: 0 when you want it to fail at once, for example in tests.
UndefinedFunctionError on a remote plain function
Symptom. MyService.my_function(1) raises UndefinedFunctionError on a node that does not run the service, although Malla.remote(MyService, :my_function, [1]) works.
Cause. On a node that has no MyService module, Malla generates a virtual module that has proxies only for callbacks (defcb). A function defined with plain def is not in it. Also, a virtual module does not exist until the service has been discovered.
Fix. Call plain functions with Malla.remote/4 or Malla.call, or define the function as a callback (defcb). If the error happens right after connecting, wait for discovery (or use the :precompile setting, see Service Discovery). If the node has the module of its own code (the service is in your application), the call runs locally instead.
A callback returns :cont or {:cont, ...} to the caller
Symptom. MyService.my_callback(1) returns :cont (or {:cont, 5}) instead of a value.
Cause. Every module in the chain returned a continuation, so the last one had nobody to pass the call to, and its raw return went back to the caller. For your own callbacks nothing in Malla provides a base implementation.
Fix. Make sure that the module at the bottom of the chain of that callback returns a real value. Check the chain with Malla.Service.get_callbacks(MyService, :my_callback): the last entry is the one that must return a value. See Callbacks.
A related one: {:cont, my_list} with a callback of one argument that is a list calls the next module with length(my_list) arguments, and fails with UndefinedFunctionError. Wrap it: {:cont, [my_list]}.
FunctionClauseError in a callback
Cause. A defcb without a catch-all clause received arguments it does not match. The exception is raised from your plugin and propagates to the caller; the rest of the chain is not asked.
Fix. Add defcb name(_args), do: :cont as the last clause, with one _ per argument.
UndefinedFunctionError for malla_span/3, status/1, malla_request/3, ...
Cause. The service uses a feature whose plugin is not in its chain: span/info need Malla.Plugins.Tracer; Malla.Status needs Malla.Plugins.Status; Malla.Request needs Malla.Plugins.Request on both sides (the called service answers malla_request/3). When the missing callback is on the other node, the caller sees it as "internal_error", and the real error is in the log of the called node.
Fix. Add the plugin to the service: plugins: [Malla.Plugins.Request] brings in the tracer and status plugins.
Plugin order surprises
Symptom. A plugin's callback is not called, or called after another one.
Inspect.
MyService.service().plugin_chain
# [MyService, PluginA, PluginB, Malla.Plugins.Base]
Malla.Service.get_callbacks(MyService, :my_callback)
# [{{:my_callback, 1}, [{MyService, :my_callback_malla_service}, {PluginA, :my_callback}, ...]}]The chain of a callback only has the modules that define it, in the order of the plugin chain.
Rules to check (see Plugins):
- A plugin sits above the plugins it depends on, so it is called before them.
plugin_deps: [AuthPlugin]means "I run before AuthPlugin", not after. - Independent plugins are called in the order of
plugins:; the first listed is called first. - A group
[A, B, C]starts A, B, C and calls C, B, A. Groups only chain the plugins that you list inplugins:. {Plugin, optional: true}does not add the plugin: it only orders it if something else (a plugin that requires it, or theplugins:list) adds it.- A plugin that returns a final value ends the chain: plugins below it are not called. A tracer plugin must depend on
Malla.Plugins.Tracer(to be above it), otherwise the defaultmalla_span/3may shadow it. add_plugin/3puts the plugin first in theplugins:list.- The chain you see is the last one built, not necessarily the one in the source:
start_link(plugins: ...),add_plugin/3,del_plugin/2andreconfigure(plugins: ...)rebuild it, and the rebuilt chain stays loaded in the VM after the service stops.
Service stuck in :failed
Symptom. MyService.get_status() is :failed; Malla.Service.is_live?/1 is false.
Look. {:ok, info} = Malla.Service.get_service_info(MyService) and read info.last_error and info.last_error_time. The log has lines like Service MyService (MyPlugin): start error: ... and status is 'failed': .... A last_error of {:child_failed, plugin} means the supervisor of that plugin's children went down (too many restarts).
What the service does. The service process stays up. While the admin status is :active, every 5 seconds it tries to start the plugins that are not running; when they all start, the status returns to :running. Each failed attempt stops the plugins that did start (calling plugin_stop/2) and sets :failed again, so service_status_changed(:failed) is called every time. With admin status :pause or :inactive a failed service is not retried.
Fix. Fix the cause that last_error names (a connection that is not there, an invalid configuration, a child that crashes in init), or, if it is a bad configuration, reconfigure/2. A service that is :failed for long is reported as not live: a Kubernetes liveness probe on /is_live restarts the pod. An exception raised in plugin_start/2 is different: it crashes the service process (and a supervisor restarts it).
{:error, {:key_not_allowed_in_service_start, key}}
Symptom. start_link/1 fails (the linked caller gets an exit; start_supervised/1 returns {:error, {{:key_not_allowed_in_service_start, key}, child_spec}}), or reconfigure/2 returns {:error, {:key_not_allowed_in_service_start, key}}.
Cause. The runtime configuration (start options, the config :my_app, MyService, ... environment of otp_app, or a reconfigure/2 update) carries one of :class, :vsn, :global, :paused, :silent, :priority, :otp_app. Those are options of use Malla.Service and cannot change at runtime.
Fix. Move the key to the use Malla.Service options, or rename your configuration key.
Malla.Cluster not connecting
- It is not started. Malla does not start
Malla.Cluster. AddMalla.Clusterto your supervision tree; without itconfig :malla, connect: [...]is never read. - Wrong key. The setting is
config :malla, connect: [...](a list of strings, or a string). The first attempt is 5 seconds after the process starts. - Wrong entry form.
"name@domain"looks for an SRV record first, then A records and connects toname@<ip>."domain"alone only looks for SRV records, with no fall back to A records. The node names have the formname@ip, so the nodes must have been started with--name(long names) and an IP in the name, for exampleRELEASE_NODE=myapp@10.1.2.3. - Cookie. All nodes must have the same cookie.
- EPMD and ports. EPMD (4369) and the distribution ports of the nodes must be reachable. SRV ports are ignored; Erlang gets the port from EPMD.
- DNS. Each lookup that fails writes
Error in resolve for ...to the log every 5 seconds. Check with:inet_res.getbyname(~c"your.domain", :a)and:srvfrom aniexon the node. - Try by hand.
Node.connect(:"myapp@10.1.2.3")shows whether it is Malla or the network:falsemeans the connection itself fails.
Other things that look like bugs
- Calls to a missing service take about 5 seconds to fail:
sna_retries(5) with one second between. Lower them for fast failure. Malla.Service.get/3raisesArgumentError("the table identifier does not refer to an existing ETS table"): the service is not running, so its table does not exist.- A callback
service_status_changed/1crashes the service with aMatchError: the chain ended with a value other than:ok. Return:cont. Malla.remote/4returns{:error, {:malla_rpc_error, {error, text}}}: the remote function raised, or the call timed out (%ErlangError{original: {:erpc, :timeout}}), or the connection broke. The remote exception is inerror; the formatted stacktrace is intext. Raise the:timeoutoption (default 15 000 ms, orconfig :malla, remote_timeout:) if it is only slow.Malla.Requestanswers"internal_error"withInternal reference NNNN: the real term is in the log of the node that produced it, next to the same number (Malla reference NNNN: ...). It is also what an error with nostatus/1clause becomes.- Changes to plugins are not seen after a restart of the service: a restarted service uses the last chain that was built; to get the compile-time one again restart the node.
reconfigure(plugins: ...)adds a plugin but itsplugin_start/2is late: the new plugins are started by the next periodic check, up to 5 seconds later.add_plugin/3does it at once.- A paused service still runs its local callbacks: pause only removes it from discovery, makes
is_ready?/1false and makesMalla.Plugins.Requestrefuse new requests.
Next Steps
- Lifecycle, Plugins, Callbacks - the rules behind the symptoms.
- Cluster Setup, Service Discovery, Remote Calls - distribution.