AI on mobile: Powering your field force – Part 5 of 5
source link: https://blogs.sap.com/2022/06/06/ai-on-mobile-powering-your-field-force-part-5-of-5/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
Introduction
Dear readers, I am a software developer, and this is my very first blog post. Generally, when we talk about a mobile app, a native android or ios app comes into our mind which requires learning separate set of languages to develop android/ios apps. But in this project we explored writing our code in PWA i.e. Progressive Web Apps.
Abstract
In the blog post https://blogs.sap.com/2022/05/26/ai-on-mobile-powering-your-field-force-part-4-of-5/ Aditi Arora has explained about tesorflow lite deployment in browser using web asembly.
In next section, I will walk you through the steps to create the UI/Frontend for Meter Reading Service with Angular and Progressive Web Apps and it’s integration to backend.
What are Progressive Web Apps?
Progressive Web Apps (PWAs) are the next big thing in web development as they bring mobile-app-like experiences to your users without requiring them to install an app from the app store/ play store. You can simply write a web application (HTML, CSS, JavaScript). This combines the best of both worlds: You deliver fast, engaging and reliable experiences whilst still having the reach and accessibility of a web page. Another added advantage of PWA’s is that it has offline support. That means complete application can work offline without the need of an internet connection. Once you are online data will be synced.
Project Overview
For this project we build our PWA with angular. This is a meter reading project (meter can be electricity, light water etc.). We have limited the scope to electricity meter for this project. A Utility Technician will use this app for navigating to the building and apartments and take picture of the meter reading, this image will be processed using Machine Learning and number from the image will be read and submitted to backend for verification. Below image shows the overview of the app.
Pre-Requisites
The following need to be installed on your machine:
- Git https://git-scm.com/downloads
- JavaScript
- HTML
- CSS
- Install the Angular CLI :- npm install –g @angular/cli
- Create a workspace and initial application :-
- ng new my–app
- cd my–app
- ng serve —open
- You can access the application at http://localhost:4200/
Please visit https://angular.io/guide/setup-local if you still have issues setting up local system for this app.
Workflow
Front End App requires 4 major steps :-
Create Login for the Utility Technician
-
- A login screen is provided for entering email and password for Utility technician.
- Login is necessary so that he can get areas/layouts allocated to him for taking meter readings.
Login
To create this screen we need to create a component named login.component.html with email and password fields. A login button will be provided which will take care of authentication from backend and upon successful login it will redirect you to dashboard page
login() {
let url = '/odata/v4/BuildingService/User?$filter=userid%20eq%20%27'+this.userid+'%27&password%20eq%20%27'+this.password+'%27';
this.restService.doHttpGetWithOptions(environment.appRoot + url, null)
.subscribe({
next: ((data: any) => {
if(data && data.value){
this.userInfo = data.value[0];
this.route.navigate(["/home"]);
}
}),
error: ((error: any) => {
console.log(error);
})
});
}Create screens for choosing a particular unit inside a layout/building
-
- User is presented with a Location screen where he can select the layout
- Then the user can select a building and then an apartment or if it’s an individual house, he can directly select it.
buildings
To create this screen we need to create 2 components named home component and buildings component. Layouts are fetched from backend using user current location. Once user selects a layout, corresponding buildings are fetched using ‘/odata/v4/BuildingService/Building’ api.
this.restService.doHttpGetWithOptions(environment.appRoot + '/odata/v4/BuildingService/Building', null)
.subscribe({
next: ((data: any) => {
if(data){
this.buildings = data.value;
}
}),
error: ((error: any) => {
console.log(error);
})
});As soon as building is selected, apartments are displayed which are fetched using ‘
/odata/v4/BuildingService/BuildingApartment?$filter=building_ID%20eq%20' + buildingId,'
this.restService.doHttpGetWithOptions(environment.appRoot + '/odata/v4/BuildingService/BuildingApartment?$filter=building_ID%20eq%20' + buildingId, null)
.subscribe({
next: ((data: any) => {
if (data) {
this.apartments = data.value;
}
}),
error: ((error: any) => {
console.log(error);
})
});Capture the meter reading for this unit.
-
- User is asked for a permission to allow camera access.
- Then user is presented with a camera and a snap button.
- User takes the meter reading, machine learning model predicts it and then user submits the predicted value with image to backend.
camera
A camera component is created with camera and snap button to allow user to capture an image.
onCameraClick(config: any) {
this.pictureTaken = false;
this.pictureEvent.emit(""+this.pictureTaken);
this.canvasRef.nativeElement.height = 0;
this.canvasRef.nativeElement.width = 0;
if (navigator.mediaDevices && navigator.mediaDevices.getUserMedia) {
navigator.mediaDevices.getUserMedia({
video: config,
audio: false
})
.then(stream => {
this.mediaStream = stream;
this.videoCapable = true;
const that = this;
this.video.srcObject = this.mediaStream;
this.video.height = this.videoHeight;
this.video.width = this.videoWidth;
this.video.play().then(() => {
that.canvas.width = this.videoRef.nativeElement.videoWidth;
that.canvas.height = this.videoRef.nativeElement.videoHeight;
});
})
.catch(err => {
//try again with removing facing mode in case of browser
if(this.videoConfig.facingMode){
this.videoConfig = {
width: this.videoWidth,
height: this.videoHeight
};
this.onCameraClick(this.videoConfig);
} else {
this.videoCapable = false;
}
});
}
}After image is taken and user is satisfied that image taken is correct, user can submit data to backend with image like shown in below code
submitData() {
// submit data to backend
//post rquest for meter reading
let d = new Date();
let dueDate = new Date();
dueDate.setMonth(d.getMonth() + 1);
this.meterReading = {
apartment_ID: this.apartmentId, billdate: formatDate(d, 'yyyy-MM-dd', 'en'),
building_ID: this.buildingId, currentreading: Number(this.currentReading),
duedate: formatDate(dueDate, 'yyyy-MM-dd', 'en'), type: "ELECTRICITY", mediaType: "application/png"
}
console.log(this.meterReading);
if (navigator && navigator.onLine) {
this.restService.post(this.meterReading, environment.appRoot + '/odata/v4/BuildingService/MeterUsage')
.subscribe({
next: ((data: any) => {
if (data) {
console.log(data);
this.bill = data;
this.restService.put(this.image, environment.appRoot + '/odata/v4/BuildingService/MeterUsage(' + this.bill.ID + ')/content')
.subscribe({
next: ((data: any) => {
this.readingSuccessful = true;
}),
error: ((error: any) => {
console.log(error);
})
});
}
}),
error: ((error: any) => {
console.log(error);
})
});
} else {
this.dataCached = true;
let meterUsageCount: number = Number(localStorage.getItem("BuildingServiceMeterUsageCount")!);
if(meterUsageCount && Number(meterUsageCount) > 0) {
meterUsageCount++;
} else {
meterUsageCount = 1;
}
localStorage.setItem("BuildingServiceMeterUsageCount", meterUsageCount.toString());
//cache for later saving
localStorage.setItem("odata/v4/BuildingService/MeterUsage/"+meterUsageCount, JSON.stringify({
'url':environment.appRoot + '/odata/v4/BuildingService/MeterUsage',
'body': this.meterReading
}));
localStorage.setItem("odata/v4/BuildingService/MeterUsage/content/"+meterUsageCount, JSON.stringify({
'url':environment.appRoot + '/odata/v4/BuildingService/MeterUsage/content',
'body': this.image
}));
}
}Provide offline functionality incase internet is not available.
-
- PWA helps us to provide offline functionality for GET Api’s. We just have to register the correponding url’s in ngsw.json file and api’s will b available offline.
- For POST Api’s a custom function is written which checks for online/offline subscription, once user is online it checks for cache and then submits the data
fetchAll() {
let finds = [];
if (localStorage.length === 0) {
return [];
}
for (let i = 0; i < this.cachedUrls.length; i++) {
try {
let key: string = this.cachedUrls[i]['url'];
let count: number = Number(localStorage.getItem(this.cachedUrls[i]['countVar'])!);
for(let j=1; j<=count; j++) {
let value = JSON.parse(localStorage.getItem(key + "/" + j)!);
if (value) {
value["key"] = key + "/" + j;
value["isNested"] = false;
if (this.cachedUrls[i]['nestedUrl'] != null) {
value["isNested"] = true;
let nestedKey = this.cachedUrls[i]['nestedUrl'];
let nestedValue = JSON.parse(localStorage.getItem(nestedKey + "/" + j)!);
if (nestedValue) {
value["nestedKey"] = nestedKey + "/" + j;
value["nestedBody"] = nestedValue['body'];
value["nestedUrl"] = nestedValue['url'];
value["nestedPart"] = this.cachedUrls[i]['nestedPart'];
value["countVar"] = this.cachedUrls[i]['countVar'];
}
}
finds.push(value);
}
}
} catch (err) {
console.log("api not cached yet!!");
}
}
return finds;
}
sync() {
let records = this.fetchAll();
if (navigator && navigator.onLine && records.length) {
records.forEach((rcdData: any, idx: any) => {
this.post(rcdData['body'], rcdData['url'])
.subscribe({
next: ((data: any) => {
if(rcdData['isNested']) {
let url = rcdData['nestedUrl'].replace(rcdData['nestedPart'], "("+data.ID+")"+rcdData['nestedPart']);
this.put(rcdData['nestedBody'], url)
.subscribe({
next: ((nestedData: any) => {
localStorage.removeItem(rcdData['key']);
localStorage.removeItem(rcdData['nestedKey']);
localStorage.removeItem(rcdData['countVar']);
records.splice(idx); //remove from records array
}),
error: ((error: any) => {
console.log(error);
})
});
} else {
localStorage.removeItem(rcdData['key']);
localStorage.removeItem(rcdData['countVar']);
records.splice(idx); //remove from records array
}
})
,
error: ((error: any) => {
console.log(error);
})
});
});
}
}Model Integration:
We had previously converted the machine learning model into web assembly format to be able to run it within the browser. Integrating it into the angular web app is done in the following steps:
- Add the artifacts to the Angular PWA assets: We need to get the transpiled web assembly model and the associated JavaScript runner. The process is already documented in [page] (https://blogs.sap.com/2022/05/26/ai-on-mobile-powering-your-field-force-part-4-of-5/) to get them. Move these into the assets’ directory of the Angular PWA.
- Make modifications to the generated JavaScript file: The generated artifacts cannot be directly embedded; we need to make a couple of minor changes:
- The module defined in the JavaScript file needs to be exported using `export` keyword. The change would look something like this:
`export var Module = typeof Module != ‘undefined’ ? Module : {};`
- Change the web assembly binary path from `wasmBinaryFile=”onnx_model_wasm.wasm”` to `wasmBinaryFile=”assets/javascript/onnx_model_wasm.wasm”`
- Code changes for consuming the artifacts:
- onnx-helper-service: This service helps in running the model. It preprocesses the image and internally calls the javascript & webassembly methods with additional parameters and gets the results.
- model-initialization-service: This service makes a call to the backend to fetch the model and cache it at the app initialization.
Conclusion:
In this blog post we learnt about how to create angular apps using PWA and how to bring offline functionality to our app. Also we learnt about how to integrate web binary to frontend app.
Please like, comment and ask questions.
References:
Credits:
I would like to thank the team Sumin Lee, Vriddhi Shetty, Gunter Albrecht, Anoop Mittal, Roopa Prabhu Nagavara, Aditi Arora, @S Deepak Chethan
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK