Spring Cloud Kubernetes For Hybrid Microservices Architecture
source link: https://piotrminkowski.wordpress.com/2020/01/03/spring-cloud-kubernetes-for-hybrid-microservices-architecture/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Spring Cloud Kubernetes For Hybrid Microservices Architecture
You might use Spring Cloud Kubernetes to build applications running both inside and outside Kubernetes cluster. The only problem with starting application outside Kubernetes is that there is no auto-configured registration mechanism. Spring Cloud Kubernetes delegates registration to the platform, what is an obvious behaviour if you are deploying your application internally using Kubernetes objects. With external application the situation is different. In fact, you should guarantee registration by yourself on the application side.
This article is an explanation of motivation to add auto-registration mechanisms to Spring Cloud Kubernetes project only for external applications. Let’s consider the architecture where some microservices are running outside Kubernetes cluster and some other are running inside it. There can be many explanations of such the situation. The most obvious explanation seems to be a migration of your microservices from older infrastructure to Kubernetes. Assuming it is still in progress, you have some microservices already moved to the cluster, while some others still running on the older infrastructure. Moreover, you can decide to start some kind of experimental cluster with only a few of your applications, until you have more experience with using Kubernetes on production. I think it is not a very rare case.
Of course, there are different approaches to that issue. For example, you may maintain two independent microservices-based architectures, with different discovery registry and configuration sources. But you can also connect external microservices through Kubernetes API with the cluster to load configuration from ConfigMap
or Secret
, and register them there allow inter-service communication with Spring Cloud Kubernetes Ribbon.
The sample application source code is available on GitHub under branch hybrid in sample-spring-microservices-kubernetes repository: https://github.com/piomin/sample-spring-microservices-kubernetes/tree/hybrid.
Architecture
For the current we may change a little architecture presented in my previous article about Spring Cloud and Kubernetes – Microservices With Spring Cloud Kubernetes. We move one of the sample microservices employee-service, described in the mentioned article, outside Kubernetes cluster. Now, the applications which are communicating with employee-service need to use the addresses outside cluster. Also they should be able to handle a port number dynamically generated on the application during startup (server.port=0
). Our applications are still distributed across different namespaces, so it is important to enable multi-namespaces discovery feature – also described in my previous article. The application employee-service is connecting to MongoDB, which is still deployed on Kubernetes. In that case the integration is performed via Kubernetes Service. The following picture illustrates our current architecture.
Kubernetes PropertySource
The situation with distributed configuration is clear. We don’t have to implement any additional code to be able to use it externally. Just, before starting client application we have to set environment variable KUBERNETES_NAMESPACE. Since, we set it to external we first need to create such a namespace.
Then we may apply some property sources to that namespace. The configuration is consisting of Kubernetes ConfigMap
and Secret
. We store there Mongo location, credentials, and some other properties. Here’s our ConfigMap
declaration.
apiVersion: v1
kind: ConfigMap
metadata:
name: employee
data:
application.yaml: |-
logging.pattern.console:
"%d{HH:mm:ss} ${LOG_LEVEL_PATTERN:-%5p} %m%n"
spring:
cloud:
kubernetes:
discovery:
all-namespaces:
true
register:
true
data:
mongodb:
database: admin
host:
192.168
.
99.100
port:
32612
The port number is taken from mongodb
Service
, which is deployed as NodePort
type.
And here’s our Secret
.
apiVersion: v1
kind: Secret
metadata:
name: employee
type: Opaque
data:
spring.data.mongodb.username: UGlvdF8xMjM=
spring.data.mongodb.password: cGlvdHI=
Then, we are creating resources inside external
namespace.
In bootstrap.yml
file we need to set address of Kubernetes API server and property responsible for trusting server’s cert. We should also enable using Secret as property source, which is disabled by default for Spring Cloud Kubernetes Config.
spring:
application:
name: employee
cloud:
kubernetes:
secrets:
enableApi:
true
client:
masterUrl:
192.168
.
99.100
:
8443
trustCerts:
true
External Registration Implementation
The situation with service discovery is much more complicated. Since Spring Cloud Kubernetes delegates discovery to the platform, what is perfectly right for internal applications, the lack of auto-configured registration is a problem for external application. That’s why I decided to implement module for Spring Cloud Kubernetes auto-configured registration for external application. Currently it is available inside our sample repository as spring-cloud-kubernetes-discovery-ext module. It is implemented according to the Spring Cloud Discovery registration pattern. Let’s begin from dependencies. We just need to include spring-cloud-starter-kubernetes, which contains core and discovery modules.
<
dependency
>
<
groupId
>org.springframework.cloud</
groupId
>
<
artifactId
>spring-cloud-starter-kubernetes</
artifactId
>
</
dependency
>
Here’s our registration object. It implements Registration
interface from Spring Cloud Commons, which defines some basic getters. We should provide hostname, port, serviceId etc.
public
class
KubernetesRegistration
implements
Registration {
private
KubernetesDiscoveryProperties properties;
private
String serviceId;
private
String instanceId;
private
String host;
private
int
port;
private
Map<String, String> metadata =
new
HashMap<>();
public
KubernetesRegistration(KubernetesDiscoveryProperties properties) {
this
.properties = properties;
}
@Override
public
String getInstanceId() {
return
instanceId;
}
@Override
public
String getServiceId() {
return
serviceId;
}
@Override
public
String getHost() {
return
host;
}
@Override
public
int
getPort() {
return
port;
}
@Override
public
boolean
isSecure() {
return
false
;
}
@Override
public
URI getUri() {
return
null
;
}
@Override
public
Map<String, String> getMetadata() {
return
metadata;
}
@Override
public
String getScheme() {
return
"http"
;
}
public
void
setServiceId(String serviceId) {
this
.serviceId = serviceId;
}
public
void
setInstanceId(String instanceId) {
this
.instanceId = instanceId;
}
public
void
setHost(String host) {
this
.host = host;
}
public
void
setPort(
int
port) {
this
.port = port;
}
public
void
setMetadata(Map<String, String> metadata) {
this
.metadata = metadata;
}
}
We have some additional configuration properties in comparison to Spring Cloud Kubernetes Discovery. They are available under the same prefix spring.cloud.kubernetes.discovery
.
@ConfigurationProperties
(
"spring.cloud.kubernetes.discovery"
)
public
class
KubernetesRegistrationProperties {
private
String ipAddress;
private
String hostname;
private
boolean
preferIpAddress;
private
Integer port;
private
boolean
register;
// GETTERS AND SETTERS
}
There is also a class that should extends abstract AbstractAutoServiceRegistration
. It is responsible for managing a registration process. First, it enables registration mechanism only if application is running outside Kubernetes. It uses PodUtils
bean defined in Spring Cloud Kubernetes Core for that. It also implements method for building registration object. The port may be generated dynamically on startup. The rest of process is performed inside the abstract subclass.
public
class
KubernetesAutoServiceRegistration
extends
AbstractAutoServiceRegistration<KubernetesRegistration> {
private
KubernetesDiscoveryProperties properties;
private
KubernetesRegistrationProperties registrationProperties;
private
KubernetesRegistration registration;
private
PodUtils podUtils;
KubernetesAutoServiceRegistration(ServiceRegistry<KubernetesRegistration> serviceRegistry,
AutoServiceRegistrationProperties autoServiceRegistrationProperties,
KubernetesRegistration registration, KubernetesDiscoveryProperties properties,
KubernetesRegistrationProperties registrationProperties, PodUtils podUtils) {
super
(serviceRegistry, autoServiceRegistrationProperties);
this
.properties = properties;
this
.registrationProperties = registrationProperties;
this
.registration = registration;
this
.podUtils = podUtils;
}
public
void
setRegistration(
int
port)
throws
UnknownHostException {
String ip = registrationProperties.getIpAddress() !=
null
? registrationProperties.getIpAddress() : InetAddress.getLocalHost().getHostAddress();
registration.setHost(ip);
registration.setPort(port);
registration.setServiceId(getAppName(properties, getContext().getEnvironment()) +
"."
+ getNamespace(getContext().getEnvironment()));
registration.getMetadata().put(
"namespace"
, getNamespace(getContext().getEnvironment()));
registration.getMetadata().put(
"name"
, getAppName(properties, getContext().getEnvironment()));
this
.registration = registration;
}
@Override
protected
Object getConfiguration() {
return
properties;
}
@Override
protected
boolean
isEnabled() {
return
!podUtils.isInsideKubernetes();
}
@Override
protected
KubernetesRegistration getRegistration() {
return
registration;
}
@Override
protected
KubernetesRegistration getManagementRegistration() {
return
registration;
}
public
String getAppName(KubernetesDiscoveryProperties properties, Environment env) {
final
String appName = properties.getServiceName();
if
(StringUtils.hasText(appName)) {
return
appName;
}
return
env.getProperty(
"spring.application.name"
,
"application"
);
}
public
String getNamespace(Environment env) {
return
env.getProperty(
"KUBERNETES_NAMESPACE"
,
"external"
);
}
}
The process should be initialized just after application startup. In order to catch startup event we prepare bean that implements SmartApplicationListener
interface. The listener method calls bean KubernetesAutoServiceRegistration
to prepare registration object and start the process.
public
class
KubernetesAutoServiceRegistrationListener
implements
SmartApplicationListener {
private
static
final
Logger LOGGER = LoggerFactory.getLogger(KubernetesAutoServiceRegistrationListener.
class
);
private
final
KubernetesAutoServiceRegistration autoServiceRegistration;
KubernetesAutoServiceRegistrationListener(KubernetesAutoServiceRegistration autoServiceRegistration) {
this
.autoServiceRegistration = autoServiceRegistration;
}
@Override
public
boolean
supportsEventType(Class<?
extends
ApplicationEvent> eventType) {
return
WebServerInitializedEvent.
class
.isAssignableFrom(eventType);
}
@Override
public
boolean
supportsSourceType(Class<?> sourceType) {
return
true
;
}
@Override
public
int
getOrder() {
return
0
;
}
@Override
public
void
onApplicationEvent(ApplicationEvent applicationEvent) {
if
(applicationEvent
instanceof
WebServerInitializedEvent) {
WebServerInitializedEvent event = (WebServerInitializedEvent) applicationEvent;
try
{
autoServiceRegistration.setRegistration(event.getWebServer().getPort());
autoServiceRegistration.start();
}
catch
(UnknownHostException e) {
LOGGER.error(
"Error registering to kubernetes"
, e);
}
}
}
}
Here’s the auto-configuration for all previously described beans.
@Configuration
@ConditionalOnProperty
(name =
"spring.cloud.kubernetes.discovery.register"
, havingValue =
"true"
)
@AutoConfigureAfter
({AutoServiceRegistrationConfiguration.
class
, KubernetesServiceRegistryAutoConfiguration.
class
})
public
class
KubernetesAutoServiceRegistrationAutoConfiguration {
@Autowired
AutoServiceRegistrationProperties autoServiceRegistrationProperties;
@Bean
@ConditionalOnMissingBean
public
KubernetesAutoServiceRegistration autoServiceRegistration(
@Qualifier
(
"serviceRegistry"
) KubernetesServiceRegistry registry,
AutoServiceRegistrationProperties autoServiceRegistrationProperties,
KubernetesDiscoveryProperties properties,
KubernetesRegistrationProperties registrationProperties,
KubernetesRegistration registration, PodUtils podUtils) {
return
new
KubernetesAutoServiceRegistration(registry,
autoServiceRegistrationProperties, registration, properties, registrationProperties, podUtils);
}
@Bean
public
KubernetesAutoServiceRegistrationListener listener(KubernetesAutoServiceRegistration registration) {
return
new
KubernetesAutoServiceRegistrationListener(registration);
}
@Bean
public
KubernetesRegistration registration(KubernetesDiscoveryProperties properties)
throws
UnknownHostException {
return
new
KubernetesRegistration(properties);
}
@Bean
public
KubernetesRegistrationProperties kubernetesRegistrationProperties() {
return
new
KubernetesRegistrationProperties();
}
}
Finally, we may proceed to the most important step – an integration with Kubernetes API. Spring Cloud Kubernetes uses Fabric Kubernetes Client for communication with master API. The KubernetesClient
bean is already auto-configured, so we may inject it. The register
and deregister
methods are implemented in class KubernetesServiceRegistry
that implements ServiceRegistry
interface. Discovery in Kubernetes is configured via Endpoint API. Each Endpoint
contains a list of EndpointSubset
that stores a list of registered IPs inside EndpointAddress
and a list of listening ports inside EndpointPort
. Here’s the implementation of register and deregister methods.
public
class
KubernetesServiceRegistry
implements
ServiceRegistry<KubernetesRegistration> {
private
static
final
Logger LOG = LoggerFactory.getLogger(KubernetesServiceRegistry.
class
);
private
final
KubernetesClient client;
private
KubernetesDiscoveryProperties properties;
public
KubernetesServiceRegistry(KubernetesClient client, KubernetesDiscoveryProperties properties) {
this
.client = client;
this
.properties = properties;
}
@Override
public
void
register(KubernetesRegistration registration) {
LOG.info(
"Registering service with kubernetes: "
+ registration.getServiceId());
Resource<Endpoints, DoneableEndpoints> resource = client.endpoints()
.inNamespace(registration.getMetadata().get(
"namespace"
))
.withName(registration.getMetadata().get(
"name"
));
Endpoints endpoints = resource.get();
if
(endpoints ==
null
) {
Endpoints e = client.endpoints().create(create(registration));
LOG.info(
"New endpoint: {}"
,e);
}
else
{
try
{
Endpoints updatedEndpoints = resource.edit()
.editMatchingSubset(builder -> builder.hasMatchingPort(v -> v.getPort().equals(registration.getPort())))
.addToAddresses(
new
EndpointAddressBuilder().withIp(registration.getHost()).build())
.endSubset()
.done();
LOG.info(
"Endpoint updated: {}"
, updatedEndpoints);
}
catch
(RuntimeException e) {
Endpoints updatedEndpoints = resource.edit()
.addNewSubset()
.withPorts(
new
EndpointPortBuilder().withPort(registration.getPort()).build())
.withAddresses(
new
EndpointAddressBuilder().withIp(registration.getHost()).build())
.endSubset()
.done();
LOG.info(
"Endpoint updated: {}"
, updatedEndpoints);
}
}
}
@Override
public
void
deregister(KubernetesRegistration registration) {
LOG.info(
"De-registering service with kubernetes: "
+ registration.getInstanceId());
Resource<Endpoints, DoneableEndpoints> resource = client.endpoints()
.inNamespace(registration.getMetadata().get(
"namespace"
))
.withName(registration.getMetadata().get(
"name"
));
EndpointAddress address =
new
EndpointAddressBuilder().withIp(registration.getHost()).build();
Endpoints updatedEndpoints = resource.edit()
.editMatchingSubset(builder -> builder.hasMatchingPort(v -> v.getPort().equals(registration.getPort())))
.removeFromAddresses(address)
.endSubset()
.done();
LOG.info(
"Endpoint updated: {}"
, updatedEndpoints);
resource.get().getSubsets().stream()
.filter(subset -> subset.getAddresses().size() ==
0
)
.forEach(subset -> resource.edit()
.removeFromSubsets(subset)
.done());
}
private
Endpoints create(KubernetesRegistration registration) {
EndpointAddress address =
new
EndpointAddressBuilder().withIp(registration.getHost()).build();
EndpointPort port =
new
EndpointPortBuilder().withPort(registration.getPort()).build();
EndpointSubset subset =
new
EndpointSubsetBuilder().withAddresses(address).withPorts(port).build();
ObjectMeta metadata =
new
ObjectMetaBuilder()
.withName(registration.getMetadata().get(
"name"
))
.withNamespace(registration.getMetadata().get(
"namespace"
))
.build();
Endpoints endpoints =
new
EndpointsBuilder().withSubsets(subset).withMetadata(metadata).build();
return
endpoints;
}
}
The auto-configuration beans are registered in spring.factories
file.
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.cloud.kubernetes.discovery.ext.KubernetesServiceRegistryAutoConfiguration,\
org.springframework.cloud.kubernetes.discovery.ext.KubernetesAutoServiceRegistrationAutoConfiguration
Enabling Registration
Now, we may include already created library to any Spring Cloud application running outside Kubernetes, for example to the employee-service. We are using it together with Spring Cloud Kubernetes.
<
dependency
>
<
groupId
>org.springframework.cloud</
groupId
>
<
artifactId
>spring-cloud-starter-kubernetes-all</
artifactId
>
</
dependency
>
<
dependency
>
<
groupId
>pl.piomin.services</
groupId
>
<
artifactId
>spring-cloud-kubernetes-discovery-ext</
artifactId
>
<
version
>1.0-SNAPSHOT</
version
>
</
dependency
>
The registration is still disabled, since we won’t set property spring.cloud.kubernetes.discovery.register
to true
.
spring:
cloud:
kubernetes:
discovery:
register:
true
Sometimes it might be usable to set static IP address in configuration, in case you would have multiple network interfaces.
spring:
cloud:
kubernetes:
discovery:
ipAddress:
192.168
.
99.1
By setting 192.168.99.1
as static IP address I’m able to easily perform some tests with Minikube node, which is running on VM available under 192.168.99.100.
Manual Testing
Let’s start employee-service locally. As you see on the screen below it has succesfully load configuration from Kubernetes and connected with MongoDB running on the cluster.
After startup the application has registered itself in Kubernetes.
We can view details of employee
endpoint using kubectl describe endpoints
command as shown below.
Finally we can perform some test call, for example via gateway-service running on Minikube.
$ curl http:
//192
.168.99.100:31854
/employee/actuator/info
Since Spring Cloud Kubernetes does not allow discovery across all namespaces for Ribbon client, we should override Ribbon configuration using DiscoveryClient
as shown below. For more details you may refer to my article Microservices With Spring Cloud Kubernetes.
public
class
RibbonConfiguration {
@Autowired
private
DiscoveryClient discoveryClient;
private
String serviceId =
"client"
;
protected
static
final
String VALUE_NOT_SET =
"__not__set__"
;
protected
static
final
String DEFAULT_NAMESPACE =
"ribbon"
;
public
RibbonConfiguration () {
}
public
RibbonConfiguration (String serviceId) {
this
.serviceId = serviceId;
}
@Bean
@ConditionalOnMissingBean
public
ServerList<?> ribbonServerList(IClientConfig config) {
Server[] servers = discoveryClient.getInstances(config.getClientName()).stream()
.map(i ->
new
Server(i.getHost(), i.getPort()))
.toArray(Server[]::
new
);
return
new
StaticServerList(servers);
}
}
Summary
There are some limitations related to discovery with Kubernetes. For example, there is not build-in heartbeat mechanism, so we should take care of removing application endpoints on shutdown. Also I’m not considering security aspects related to allowing discovery across all namespaces and allowing an access to API for external applications. I’m assuming you have guarantee the required level of security when building your Kubernetes cluster, especially if you decide to allow external access to the API. In fact, API is still just API and we may use it. This article shows an example of use case, which may be useful for you. If you compare it with my previous article about Spring Cloud Kubernetes you see that with small configuration changes you can move application outside cluster without adding any new components for a discovery or a distributed configuration.
Recommend
-
11
Microservices With Spring Cloud Kubernetes Spring Cloud and Kubernetes are the popular products applicable to various different use cases. However, when it comes to microservices architectur...
-
7
Running Presto in a Hybrid Cloud Architecture ·Migrating SQL workloads from a fully on-premise environment to cloud infrastructure has numerous benefits, including alleviating resource contention and reducing costs by paying for computation r...
-
8
How to implement a cloud-first strategy with hybrid multicloud architecture
-
8
Frank Luyckx September 1, 2021 5 minute read
-
3
How to Optimize Your Hybrid Cloud ArchitectureEnterprises increasingly rely on a hybrid cloud strategy to deliver scalability, flexibility, configurability, and control. But for many, cloud spend is wasted money. Here’s h...
-
8
Quarkus for Spring Developers: Kubernetes-native Design Patterns ...
-
4
What Is Hybrid Cloud? December 1, 2021 by Molly Clancy // 9 Comments
-
2
Interview with Leonid Sandler, co-founder and CTO of ARMO “Kubernetes is just a reflection of the microservices and container architecture it manages.”
-
6
Hybrid Cloud: Flexible Architecture for the Future of Financial Services
-
3
Designing Sustainable Hybrid Cloud Architecture: The Crucial Role of Carbon Footprint as a Non-Functional Requirement ...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK