This blog is part of a blog series, so you can find the first page here (https://blogs.sap.com/2023/02/02/sap-cpi-ci-cd-from-from-zero-to-hero/). This is the agenda we’re following:

• Backup Binaries and Source Code
• Code inspection and quality control
• Release management
• Certificates expiration (explained in this page)
• Automated Testing
• Documentation
• Code Review

Certificates expiration

Certificates are one of the most powerful mechanisms that you can use to secure your applications. One of the key aspects that also contributes to security is that certificates are time bounded artifacts that do have an expiration date. Expiration dates can be different from certificate to certificate (according to CA rules, if you use a self signed generated certificate, etc).

Now imagine that you work with Cloud Integration and you need to communicate with many different parties: some external servers belonging to your partners, some cloud applications, others on-premise controlled by yourself. If any of the certificates associated with the servers you’re using have expired most likely your interface would be impacted and would stop working as well.

Therefore, wouldn’t it be cool if you could know with some time upfront that a certificate is about to expire and that you should renew it or contact your vendor to renew it?

This was a must for us, fortunately when I was setting up CALM for our operations, there’s a health monitoring section that allows you to validate your certificates expiration dates allowing to associate notifications reporting it to your team. On top of that it would allow you to configure different thresholds levels with different actions, allowing you to notify different teams as the deadline approaches for instance

That was exactly what we needed, so I followed the setup on that and did a test of setting up an about to expire certificate. In the end, CALM was not working as we expected, so we decided to go with our custom jenkins pipeline instead. Below you can find the architecture we followed. Next I’ll detail all the steps that influenced us to give up on CALM and go with the custom pipeline.

Certificates custom pipeline steps

CALM Health Monitoring – 1st attempt

Setup only my email as the email to be notified and started to receive empty emails on a 5m basis. I realized most likely was what I just configured. Did some research, couldn’t find the route cause and opened an incident with SAP. The answer: a bug, that would be fixed in a short time. I’ve waited until they told me it was ready to use again.

Empty email coming with no subject and nothing relevant on the body

CALM Health Monitoring – 2nd attempt

Now I don’t receive an empty email, but instead an email mentioning that there are certificates approaching expiration date with a link to CALM. Not optimal, from my perspective since I had to open the link to really see what are the certificates about to expire. Nevertheless I thought, perhaps there’s a way to configure the body of this email by using placeholders, so that was no show stopper.

The show stopper was that I was receiving the same emails every 5m (the time CALM data collector runs to extract data from the cloud integration tenants). That was definitely a no-go, I reported, the team acknowledge that this could be a problem but there was no plan to fix it in a short time

Expired certificates with a link to CALM

Jenkins approach

So we decided to use a custom Jenkins pipeline approach, composed of 4 steps that you can find on the first diagram of this page. We run this pipeline for each of our 4 environments so that we can cover everything.

Step 1 – Download Cloud Foundry service keys using certificates and check expiration date

For this cloud foundry already provides a very easy to use api that will return you all the service keys, so we just needed to evaluate expiration dates for them. Example of the code below:

///
/// Loads a list of all service keys with certificates available on an cloud foundry (CF) runtime and their respective expiration date
///
def loadCFServiceKeyCertificatesExpirationDates(String token,String cfHost)
{
	def certs = [:] 
	try {
		def serviceKeysListResponse = httpRequest acceptType: 'APPLICATION_JSON', customHeaders: [[maskValue: false, name: 'Authorization', value: token]], 
													  ignoreSslErrors: true, validResponseCodes: '100:399, 404', timeout: 30, 
													  url: 'https://'   +   cfHost   +   '/v2/service_keys';
		if (serviceKeysListResponse.status == 404)  {
			//invalid Flow ID 
			error("Unable to list SAP CF service keys certificates expiration dates");
		} else {
            def serviceKeyJson = new JsonSlurper().parseText(serviceKeysListResponse.content ) 
			// aux list is being created to prevent jenkins issues with global objects not serializable 
			serviceKeyJson.resources.each {
                if (it.entity != null){
                    def serviceKeyName = it.entity.name
                    def certificateValue = it?.entity?.credentials?.oauth?.certificate
                    if(certificateValue!=null && !"".equals(certificateValue))
                    {
                        String certB64 = certificateValue.replace("-----END CERTIFICATE-----", "").replace("-----BEGIN CERTIFICATE-----", "").replaceAll("[\\n\\t ]", "").trim()
                        byte[] encodedCert   = Base64.getDecoder().decode(certB64);
                        ByteArrayInputStream inputStream  =  new ByteArrayInputStream(encodedCert);

                        CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
                        X509Certificate cert = (X509Certificate)certFactory.generateCertificate(inputStream);
                    
                        Date certExpirydate = new SimpleDateFormat("yyyy-MM-dd").parse(cert.notAfter.toTimestamp().toString());
						def subjectDN = cert.getSubjectDN().toString()
                        certs[serviceKeyName] = [certExpirydate:certExpirydate,subjectDN:subjectDN]

                    }
                }
			}

		}

	} catch (Exception e) {
		error("Retrieving CF service keys:\n${e}") 
	}
	return certs;
}

Step 2 – Download Cloud Foundry service keys using certificates and check expiration date

Next, since we had already a pipeline downloading all our certificates and backing them up, it was very easy to use it to extract the expiration dates for the list of certificates stored on cloud integration.

Example of the code below:

///
/// Loads a list of all certificates available on an CPI environment
///
def loadAllNonSAPCertificates(String token,String cpiHost)
{
	def certs = [:] 
	try {
		def certificateListResponse = httpRequest acceptType: 'APPLICATION_JSON', customHeaders: [[maskValue: false, name: 'Authorization', value: token]], 
													  ignoreSslErrors: true, validResponseCodes: '100:399, 404', timeout: 30, 
													  url: 'https://'   +   cpiHost   +   '/api/v1/KeystoreEntries';
		if (certificateListResponse.status == 404)  {
			//invalid Flow ID 
			error("Unable to list SAP CPI Certificates");
		} else {
			def list = new JsonSlurper().parseText(certificateListResponse.content ) 
			// aux list is being created to prevent jenkins issues with global objects not serializable 
			list.d.results.each {
				if (it.Owner != "SAP"){
                    certs[it.Alias]  = [certExpirydate:it.ValidNotAfter,subjectDN:it.SubjectDN] ;
                }
			}

		}

	} catch (Exception e) {
		error("Retrieving CPI Certificates:\n${e}") 
	}
	return certs;
}

Step 3 – Check expiration date for the SSL certificate being used on our on premise CI/CD server

We had setup SSL for our Jenkins and Gitea servers running on our on-premise server. This certificate also expires and also needs to be renewed, so we could check that expiration as well

Example of the code below:

def days = 45
try{
		println("Checking if gitea/jenkins certificate is expired on CI/CD server")
		def thresholdAlertSeconds = 24*3600*days
		def result = bat "openssl x509 -checkend " + thresholdAlertSeconds + " -noout -in \"C:/WorkSpaces/Certificates/certificate.pem\""
}catch(Exception exc){
		isToSendEmail = true
		notifBody += '<tr><td>Jenkins/Gitea certificate</td><td> </td><td>Expires in less than '+days+' days</td></tr>'
}

Step 4 – Send email notification

Final step is to send the final html containing all the certificates that are expired or about to expire to your Integration and Basis teams. Example of that code below:

if(isToSendEmail){
	notifBody += '</table>'
	unstable('Certificates expired or about to expire found!')
	emailext mimeType: 'text/html', body: '${JELLY_SCRIPT,template="html"}' + notifBody, 
		from: "[email protected]", 
		to: '[email protected];[email protected]',
		subject: 'Build unstable in Jenkins for Environment ['+environment+']: $PROJECT_NAME - #$BUILD_NUMBER'
}

As an example output, you can see that we identify the certificate as well as the CN and the date of expiration and we report it on a table

Example of email notification

CALM Health Monitoring – An update

We stopped following up on CALM for certificates expiration, but the last time I’ve checked, now the email looks much better as you can see below.

CALM email notification update

Summary

In this topic, I’ve introduced the checks we’re doing in regards to certificates about to expire. If you’re starting now with this topic and you have CALM license, I would advise you to try that out first and see if it already fits your needs. If not, you can always rely on a similar pipeline to what we’re using.

I would also invite you to share some feedback or thoughts on the comments sections. I’m sure there are still improvement or ideas for new rules that would benefit the whole community. You can always get more information about cloud integration on the topic page for the product.