In this post, we will share our experience in developing a custom Flutter plugin for Veriff’s mobile products.

Flutter is Google's UI toolkit for building beautiful, natively compiled applications for mobile, web, desktop, and embedded devices from a single codebase.

Flutter plugins are Dart packages that contain an API written in Dart language which combines multiple platform-specific implementations. Plugins can be written for Android (Kotlin or Java), iOS (Swift or Objective-C), web, macOS, Windows or Linux, or any combination of these.

Flutter packages are modules that contains only Dart code. Flutter plugins on the other hand, are  packages with a difference. Plugins contain both Dart and platform specific code (Kotlin, Swift, etc.). You can create a Flutter plugin with the following command:

Please refer to Flutter’s official documentation for more information about creating a Dart package.

A Flutter plugin structure has these folders:

lib - contains Flutter code in .dart files

ios - contains iOS platform code written in Swift or Objective C

android - contains Android platform code written in Kotlin or Java

example - an example Flutter app that could demo the usage of the plugin; the example app is optional but highly recommended as per community guidelines

Apart from these folders there will also be some mandatory files like:

README.md - a readme file with details about the plugin

pubspec.yaml - a yaml file containing metadata about the plugin, the information in this file will be used while publishing also

Communication between Flutter and native platforms 

The Flutter plugin will obviously need to communicate with the corresponding native platform. Instead of using code generation or reflection for communication, Flutter uses a simple message passing through a channel to achieve communication to and from native platforms. You can use a MethodChannel from Flutter side to communicate with the native platforms which could be received using a FlutterMethodChannel in iOS and a MethodChannel in Android. An important thing to note is that even though Flutter sends message calls asynchronously, the channel method invocations must be done in the platform's main thread.

Alternately if you are comfortable with using only Objective-C for iOS and Java for Android you could also consider using the Flutter package Pigeon for sending structured type-safe messages using code-generation.

The Flutter MethodChannel uses a standard message codec and efficiently sends simple JSON-like values like int, double, string, list, map etc. The full list of supported types and their respective converted platform types can be found here. 

The supported list of data types and their equivalent platform conversion at the time of writing is as below.

For debugging purposes Flutter provides tools.

You can use flutter doctor to check all installed tooling for Flutter or flutter analyze -d <DEVICE_ID> to analyze your source code. Keep in mind that you should replace DEVICE_ID with the device’s ID that you are debugging with.

You can run the example app with your plugin by running this command:

Depending on the type of device (Android/iOS) connected to your computer it will run the corresponding platform version.

Once your plugin is ready and tested it should be published to pub.dev so that other developers can use it in their projects. Before publishing, you have to make sure that your pubspec.yaml, README.md and CHANGELOG.md files are filled correctly. Some fields to double check in the pubspec.yaml file are: 

name: This will be the name of your plugin that 3rd party developer will be using to install your plugin

description: A human readable short description about your plugin

environment: Since Dart 2 was released, the environment value specifies the Dart and Flutter version the plugin depends on, using the sdk and flutter child values

Detailed descriptions about other fields can be found here 

In order to publish the plugin you need a developer account in pub.dev. You can make a personal account or you can make a verified account for a domain. The steps to create a verified account can be found here. Once you have filled in the required extra files and made a developer account you are ready to publish.

One important aspect about publishing a plugin is that publishing is non-reversible. This is to make sure that the developers who are depending on your builds do not have to deal with a broken build if you un-publish your plugin. Hence pub.dev policy does not allow un-publishing your plugins. 

Since publishing is non-reversible you need to do a dry run of the publish to see that everything is ready for publishing, to check if everything is correct without actually needing to publish. You can do a dry run by running this command:

Once you have made a dry run and confirmed everything is correct you can do the actual publishing by running the publish command:

Once you run the publish command flutter will log a link in your terminal to login to your pub.dev account, and you’ll need to click that link and complete authentication via your browser. Once logged in and authorised, your plugin will be published and in 10-20 mins will be accessible for third party developers.

🎉  Congrats you have published your Flutter plugin

Once you have published your plugin you can see the plugin score it got from the plugin page in pub.dev. The plugin score is computed considering different criteria like documentation, following dart conventions etc. Along with the score, you will also get pointers on how to improve the score. It’s highly recommended that you try to improve your score as much as possible.

You might want to automate publishing your plugins especially from your CI/CD pipeline. In order to do that you need to create a file in the following path in your CI/CD environment ~/.pub-cache/credentials.json with the following content:

Replace accessToken, refreshToken, tokenEndpoint and expiration with your corresponding values. 

You can find the  values in the  ~/.pub-cache/credentials.json file in your computer after you publish for the first time. And yes, unfortunately this means you have to do the first publish manually from your computer to get them.

Once you have made the credentials.json file in your CI environment you can run the following command from your CI/CD to make the publishing automatically without the prompt to login manually:

Vivekanandan A S - Senior Android Engineer

We delve into how Veriff created our custom Flutter plugin for our different mobile products which we offer our clients.

None

Vivek, a Senior Android Engineer, and Mert, our SDK Team Manager, delve into how Veriff created our custom Flutter plugin for our different mobile products which we offer our clients.

A technical blog looking at how Veriff created our custom Flutter plugin for our different mobile products which we offer our clients.

Developing a Flutter Plugin for Veriff's Mobile Products

Technical

Engineering

Due to the ever increasing importance of AI in our stack, we are extensively using graphics processing units (GPUs) in Kubernetes to run various machine learning (ML) workloads. In this blog, I’ll describe how we have been sharing GPUs between pods for the last 2 years to massively reduce our infrastructure cost.

When using NVIDIA GPU's, you need to use the NVIDIA device plugin for Kubernetes that declares a new custom resource, nvidia.com/gpu , which you can use to assign GPUs to Kubernetes pods. The issue with this approach is that you cannot split them between multiple applications (PODs) and GPU's are a very expensive resource.

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: gpu-example
spec:
  restartPolicy: OnFailure
  containers:
  - name: gpu-example
    image: eritikass/gpu-load-test
    imagePullPolicy: Always
    resources:
      limits:
        nvidia.com/gpu: 1
```

So what happens when you deploy a GPU using applications like this? Kubernetes is using the nvidia.com/gpu resource to deploy this pod into a node where a GPU is available.

If you connect (ssh) into the node where this pod is running, and use nvidia-smi command there, you can get a result similar to this.

The info visible here is that our node has 1 NVIDIA Tesla T4 GPU with 15109MiB of memory and we are using 104MiB of that with one process (our deployed pod). 

Internally there is one very important variable that is given to each pod and the application is using this to know which GPUs to use.

# connect to pod
kubectl exec -it pod/gpu-example -- bash

#and check
echo $NVIDIA_VISIBLE_DEVICES

In the example above, this GPU - GPU-93955ff6-1bbe-3f6d-8d58-a2104edb62db - is being used by the application. When you have a node with multiple GPUs, then actually all pods can access all GPUs - but they all are using this variable to know which GPU they should access.

This variable also has one "magical" value: all. Using this you can override GPU allocation and let your application know that it can use any (all) GPUs present in your node.

We know that our example application is using 104MiB and the GPU has a total 15109MiB, so in theory we can fit it 145 times into the same GPU.

In the next example you need to have AWS EKS cluster with 1 GPU instance (g4dn.xlarge), and with some modifications it should be possible to use it in any Kubernetes cluster with Nvidia GPU nodes available.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-example
spec:
  # In this example lets run 5 pods for demo purposes.
  #
  # NB: aws g4 nodes can run maxium of 29 pods due networking limits,
  # https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI
  replicas: 5
  selector:
    matchLabels:
      app: gpu-example
  template:
    metadata:
      labels:
        app: gpu-example
    spec:
      # Lets use affinity to make sure pod(s) in this deployment
      # can be only assigned to nodes that are using g4dn.xlarge instance.
      # This way we can be sure there is GPU available.
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - g4dn.xlarge
      containers:
      - name: gpu-example
        image: eritikass/gpu-load-test
        env:
          # this will let cuda application know to use any GPU available
          - name: NVIDIA_VISIBLE_DEVICES
            value: all
```

If we check after that we can see that all 5 pods are assigned to the same node.

And when we check nvidia-smi in this node we can see there are 5 GPU processes running from those pods.

Now when we know that multiple GPU-using pods can run in the same node, we need to make sure they are split between nodes based on the node's capacity to handle the pod’s requirements. For this we can use Custom Resources to let all GPU nodes know that they have so much GPU memory available to use. 

We will be using DaemonSet to add custom GPU memory resource (we named it veriff.com/gpu-memory) to add nodes with an NVIDIA GPU attached. DaemonSet is a special type of Kubernetes deployment that will run 1 pod in some (or all) cluster nodes, it is quite often used to deploy things like log collectors and monitoring tooling. Our DaemonSet will have nodeAffinity to make sure it will only run in  g4dn.xlarge nodes.

```yaml
---
#
# config maps that is holding the script that is run inside DaemonSet in all gpu nodes to set gpu memory
#
apiVersion: v1
kind: ConfigMap
metadata:
  name: add-gpu-memory
  namespace: kube-system
data:
  app.sh: |
    #!/bin/bash
    gpu_memory_value="15079Mi"
    timeout 240 kubectl proxy &
    sleep 3
    curl --header "Content-Type: application/json-patch+json" \
        --request PATCH \
        --max-time 10 --retry 10 --retry-delay 2 \
        --data "[{\"op\": \"add\", \"path\": \"/status/capacity/veriff.com~1gpu-memory\", \"value\": \"${gpu_memory_value}\"}]" \
        "http://127.0.0.1:8001/api/v1/nodes/${K8S_NODE_NAME}/status"
    echo " * ${gpu_memory_value} of gpu memory added to ${K8S_NODE_NAME}  (veriff.com/gpu-memory)"
    sleep infinity

---
#
# this DaemonSet will run in all g4dn.xlarge nodes and patching them to add them gpu memory custom resources
#
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: add-gpu-memory
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: add-gpu-memory
  template:
    metadata:
      labels:
        name: add-gpu-memory
    spec:
      tolerations:
        - key: "special"
          operator: "Exists"
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: node.kubernetes.io/instance-type
                operator: In
                values:
                - g4dn.xlarge
      serviceAccountName: add-gpu-memory
      containers:
        - name: add-gpu-memory
          image: bitnami/kubectl
          resources:
            limits:
              cpu: 40m
              memory: 50M
            requests:
              cpu: 1m
              memory: 1M
          volumeMounts:
            - mountPath: /app.sh
              name: code
              readOnly: true
              subPath: app.sh
          command:
            - bash
            - /app.sh
          env:
            #
            # this variable is used in script to patch nodes to know what node he is running
            #
            - name: "K8S_NODE_NAME"
              valueFrom:
                fieldRef:
                  apiVersion: "v1"
                  fieldPath: "spec.nodeName"
      priorityClassName: system-node-critical
      volumes:
        - name: code
          configMap:
            name: add-gpu-memory
#
# rbac permissions used by daemonset to patch nodes
#
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: add-gpu-memory
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: add-gpu-memory
  namespace: kube-system
rules:
  - apiGroups:
    - "*"
    resources:
    - nodes
    verbs:
    - get
    - list
  - apiGroups:
    - "*"
    resources:
    - nodes/status
    verbs:
    - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: add-gpu-memory
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: add-gpu-memory
subjects:
  - kind: ServiceAccount
    name: add-gpu-memory
    namespace: kube-system
```

If you check your node using “kubectl describe node/NAME” you can see it has veriff.com/gpu-memory resource available.

Now when we know that our gpu memory adding script works, let's scale our cluster up so we have multiple GPU nodes available.

In order to test it out, we will modify our deployment to use this new resource.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpu-example
spec:
  replicas: 6
  selector:
    matchLabels:
      app: gpu-example
  template:
    metadata:
      labels:
        app: gpu-example
    spec:
      containers:
      - name: gpu-example
        image: eritikass/gpu-load-test
        env:
          # this will let cuda application know to use any GPU available
          - name: NVIDIA_VISIBLE_DEVICES
            value: all
        resources:
          requests:
            veriff.com/gpu-memory: 104Mi
          limits:
            veriff.com/gpu-memory: 104Mi
```

After that it is visible that pods are split between nodes.

With regular resources like CPU and memory, kubernetes will know how much pods (containers) are actually using and if anyone tries to use more - kubernetes will restrict it. However with our new custom resource, there is no actual safeguard in place that some “evil” pod can’t take more GPU memory than declared. So you need to be very careful when setting the resources there. When pods try to use more GPU memory than a node has available, it usually results in some very ugly crashes.

In Veriff, we resolved this issue with extensive monitoring and alerting for our new GPU Custom Resources usage. We have also developed tooling in-house to measure ML applications GPU usage under load.

All code examples from this post can be found in https://github.com/Veriff/gpu-sharing-examples.

Siim Tiilen, a Quality Engineer in our DevOps team, explains practically how Veriff shares GPUs between pods and how it's helped us reduce our overall infrastructure cost.

A blog looking at how Veriff shares GPUs between pods for the last 2 years to massively reduce our infrastructure cost.

How Veriff Shares GPUs - A technical guide

This article will not be filled with code snippets, or awesome logic around how to optimise, or examples of how we built and customised the hell out of Kubernetes for GPU sharing.

Instead, I'll shed some light on the role in Veriff we call Technical Support. 

The job title is quite straight forward. Technical support is surely a group of techy people, doing support. Simple right? Well, not quite. Often people think of support with certain undertones, and even more often tech support and customer support are all just seen as the same support. 

In Veriff, we have an excellent Customer Support team as well. They are responsible for external communication and keeping our customers happy in case of any issue or query.

In our organisation CS is currently part of our Revenue division and Technical Support is in our Engineering division. 

The main job of our technical support team is to save time for development teams, so they can focus on what they do best. This is done by being an escalation point for our CS and other internal teams. There are often requests coming from customers where we have functionality but no tooling yet with UI. So technical support is there for these hacky solutions. Not to mention any technical question there might be about our product, features or upcoming developments. Obviously we research and debug all issues possible and in case bugs are found we escalate. Statistically speaking we escalate ~7.5% of all our tickets. Mainly the tasks are exactly as explained above where we have scripts to manipulate things we need. Additionally by keeping statistics around all the requests we can easily identify the demand for tooling. This makes technical support a good source of knowledge for product owners if the statistics are done right. 

The next key role for Technical Support is Incident Management. 

Technical support is on the frontline for all incidents in Veriff. We have OnCall rotation and react to all incidents 24/7. It can be a full system failure or the slightest change in a mistake rate. Technical support is here to pick up the issue, do the impact assessment, and involve the correct stakeholders. Sometimes we can simply restart the failing pod in the middle of the night or remove a poisoned message from rabbitMQ without the need to wake up our developers. 

Incidents can be escalated automatically by system monitoring or manually by anyone.

We also make sure that our customers are updated in a timely manner about the current state of the issue. We then drive the incident to resolution and make sure there is a post mortem available to everyone. I’ll go into more detail about how we manage incidents in Veriff in a future blog. 

These are easily our two most important tasks, but of course there’s more to our daily work. 

Technical support is also here to automate stuff and make everyone’s life better while working here. 
We look out for possibilities to reduce manual work for any department in Veriff, create visibility where it’s lacking, and sometimes simply to say /Thanks to someone in Slack. 

It is also the most motivating part of this role, especially for those who see themselves becoming a software engineer in future. You can learn, research and have simple enough engineering tasks (with real value to the company) where you can put new skills into practice! 

Here are a few interesting examples of the things we’ve done here: 

Automating communication between two ticketing systems - Jira and Intercom <-> comments. To make sure our comments are easy-to-use for Customer Support and vice versa. The existing integration didn’t meet our custom requirements.

Resource reporting for engineering and product collaboration using Jira and Google Sheets - As we don’t follow the built in Jira logic with dev points but man hours instead, we needed to have a resource availability map for product and engineering to be in better sync on available resources.

Team budget visibility tool for PeopleOps - We have many small teams here and budgeting for team events is hard to manage. So we created an integration with BambooHR and Google sheets for all team leads to know the exact state of their available budget. 

There are a lot more examples of these integrations, and other things technical support do which all add value to our organisation. 

To sum things up, I’d say the Technical Support in Veriff is more of a hybrid - somewhere between Devops, Developers, Q&A, Customer Support and People. Veriff’s own answer to legendary secret agent, MacGyver.

Our Technical Support Team Lead, Ott Ristikivi, delves into what the role of 'Technical Support' actually means within Veriff, what big challenges they face daily, and why it's probably not what you expect.

A blog from our Technical Support Team Lead, Ott Ristikivi, looking at the role of Technical Support within Veriff.

It’s no secret that organizations across the globe are eager to harness data. It’s also no secret that most organizations do a very poor job of it. It’s an uphill battle with a multitude of challenges, both technical and operational. 

Some of the technical challenges include building a platform that can handle a large number of concurrent queries, developing robust and reliable data pipelines, and wrangling data in a variety of standard and exotic formats. 

On the operational side, agreeing on the definition of key metrics, teaching your colleagues to use the tools and data provided by your data team, and establishing data stewardship are some of the first hurdles you’ll need to overcome. In addition, adhering to data privacy regulations such as GDPR without compromising the integrity of your data falls under both categories.

Analytics at Veriff started with a very simple architecture. It started with a database. The database was a read-only replica of our production database. This was suboptimal for several reasons.

Firstly, this database was neither designated nor designed to support complex analytical queries. The data model was highly normalized, because the source database was designed to support Online Transactional Processing (OLTP). This is wonderful if you have a large number of concurrent transactions, all writing or updating a small amount of data, but suffers when trying to read large amounts of data, especially if any processing or aggregating is involved. To make matters worse, this database was shared by all engineers and developers. 

Secondly, being a read-only database, we didn’t have the ability to persist results of long-running queries. We also couldn’t set up materialized views on the source database, because long running queries could interrupt availability of our service. Generally, that’s not considered to be desirable. 

Finally, we had limited tooling on top of this replica. That means, if you wanted data, you had to get your hands dirty with SQL. While all of our analysts and developers were comfortable with this, the most demand for data came from product owners and managers that weren’t familiar with SQL. This created a hard dependency on analysts, who now had to spend a large amount of time retrieving data and less time extracting value and insights from data.

The first issue we wanted to address was the lack of tooling. Analyst time is too scarce to spend on frequent, often repeating, data pulls. If we could provide an interface between the database and the user that allowed them to pull the data themselves, this should allow analysts to focus their efforts elsewhere.

After testing a few tools, we settled on Looker - a tool that provides a semantic layer, an intuitive UI, and an extensive visualization component. The semantic layer allowed us to map objects in our database to corresponding objects in Looker. That way, users could select specific columns from specific tables in a point-and-click manner. On top of that, we could also map arbitrary SQL snippets and queries to Looker dimensions, allowing us to implement pieces of business logic such as metrics definitions in a repeatable, efficient way. Analysts were still responsible for developing and maintaining the semantic layer, but it allowed them to do so in a centralized, version-controlled environment. Once we had mapped our database to Looker, business users were able to start querying data themselves and build their own dashboards. 

However, there were still significant issues with this setup. Looker was sitting on top of the same production replica, and all the shortcomings of that database as an analytics source were unchanged. As a matter of fact, it made some of the issues more pronounced. Now, instead of analysts and developers being the only people running heavy queries, anyone with access to Looker was able to do so.

Around the same time we introduced Looker, Veriff had hired a team of experienced database architects. One of the first things they did was set up a more sophisticated replication system to ensure that databases used by production services were isolated from other workloads, and teams that required a replica of the database for a specific function could have one. 

As part of this initiative, a replica database was set up specifically for analytics. This was a significant improvement for a few reasons. Principally, it resolved the second issue brought up at the beginning of this post - the inability to persist the results of complex queries. The analytics replica was not a read-only database. We were able to write the output of queries as tables that could then be used further down the line. This both increased the performance of the database, and allowed analysts to build tables that expressed our business logic in a more denormalized fashion rather than relying on the highly normalized data model underneath.

The new setup was already a remarkable improvement on our initial setup, but we still needed to take it further. We were still facing a multitude of challenges. Veriff was becoming more data literate, and requests were becoming more complex. This was great, but a lot of the requests required data from sources outside our database, and we didn’t support querying multiple sources from one interface. In addition, data privacy regulations like GDPR meant some data could not be stored in our analytics databases, but we still needed to keep an aggregated overview of our business that guaranteed historical integrity. 

We’ll cover how we tackled these challenges, along with some of our plans for the future, in an upcoming blog post. If you’re interested in what you read here, and want to come contribute to it, we’re hiring Data Engineers.

Nicholas Vandrey, Veriff's Business Intelligence Team Lead, writes his first blog in a series taking a deep dive into how businesses currently use data, and how to make sure you're doing it right.

Education

A blog by Veriff's Business Intelligence Team Lead, looking at how Veriff has scaled it's use of data as the business has grown and why Looker was a crucial cornerstone.

Veriff will only use the information you provide to share blog updates.

You can unsubscribe at any time. Read our privacy terms

Get the latest from Veriff. Subscribe to newsletter

Engineering

Back to Library

Developing a Flutter Plugin for Veriff's Mobile Products

How Veriff Shares GPUs - A technical guide

The Role of Technical Support in Veriff

A Practical Guide to Going Data-Driven

Get the latest from Veriff. Subscribe to newsletter

Products

Solutions

Resources

About