A Git Based Dependency Management. For LabVIEW?

What about a git based manager for your source code, zip files, binaries (PPLs), etc…?

For a while I’ve been trying to write this post, but several reasons kept me from finishing the write. One of the things is that I didn’t have a good solution for it. I had a workable solution, but it was far from being a good one, but until we get to the topic of a possible solution, let’s recap some things.

LabVIEW and SCC

The history with LabVIEW with Source Code Control Systems hasn’t been flowers.

One of the main disadvantages of LV is the binary proprietary format of the VIs, which hold the real source code. Libraries, Projects and Classes are XML format, but they are eventually holding some binary data inside.

The SCCs always tries to minimize disk footprint of the source code by working on Deltas, which is difference of the subsequent codes, where the last version is nothing but a bunch of differences from the original file.

While this works with all other textual languages, with the VIs this is not possible, Git just creates another copy of the VI on every modification, and so on, and so on…. You already know where you getting to.

Gladly well designed VI’s are usually very small, and checking “separating compiled code from source” helps, so the footprint can be greatly minimized.

SVN was the first choice for LV Users in general at the time Git was rather new, but it’s centralized nature, and the usage of “locking” files, demonstrated that a different workflow must be created, and raised questions like: Why do you need to lock the file to prevent someone editing it? Why in the first place is someone trying to edit the same file as you?

So far, Git has proven to be one of the bests for LabVIEW, not only due to its decentralized nature and feature rich application, but due to the whole ecosystem built around it.

Stil, we lack some real support on the IDE, but that’s another history.

Decentralized Projects

An easy solution for the problems with locked files or even merge conflicts is: avoid editing the same file. Obviously, right?

A good design of libraries and classes prevent that. Each developer should work on a different piece of software in a coordinated manner so conflicts can be avoided.

A creation of separated repositories are the second good solution, allowing fast git commands like push, pull (possible less files and smaller footprint), and distributing the code in a way that helps also Git to manage the files. Not to mention faster automation (CI) iterations: unit tests, static code analysis and builds.

But, with that, comes other challenges that LabVIEW hasn’t been prepared for: Dependency Management.

Versioning

This is one of the things that I most value in a good software. A reliable versioning history. Software that we can trust in version updates. Yes, we should always test after a dependency update, and that’s one of the reasons why Unit Tests are created in first place.

In previous posts, I explained all the steps for creating an automatic versioning system with nothing more than using conventional commits, based in git tags, of course. Who trusts LV with versioning classes and libraries?

Project Organization

Other important topic that I already mentioned in a previous post: Project Organization. There are many reasons why you should use a known project hierarchy: help you now, help others, help you in the future. Create a template and stick to it.

There are two folders that I most value in an organized project: Source and Libraries. The first one contains of course the main part of your code and the second one contains your dependencies. Naming is irrelevant here, what it is most important is that they are separated and in preference, they are located in “parallel” with your source file.

Why? LabVIEW linker will link it as a relative content to your source.

Don’t you believe it? Try doing it and then read the Project file (.lvproj) afterwards. Agh, ok, it is not inside the VI binary code, but from that you can have a clue on what’s happening.

Okay, I will spare you from that, see the example down here.

    <Item Name="lib" Type="Folder">
			<Item Name="Library A.lvlib" Type="Library" URL="../lib/lib-a/Library A.lvlib"/>
			<Item Name="Library B.lvlib" Type="Library" URL="../lib/lib-b/Library B.lvlib"/>
		</Item>
		<Item Name="src" Type="Folder">
			<Item Name="MainProject.lvlib" Type="Library" URL="../src/MainProject/MainProject.lvlib"/>
		</Item>

Some people call this methodo “flat” hierarchy or something like that to express how dependencies are in disk.

On the other hand, many solutions that use vi.lib and user.lib has this dependency with the LV Folder installation and working with different versions of LV and Libraries can be a challenge, unless you are really careful, and always updates your packages. But that is not always the case.

Not to mention Continuous Integration runners that generally work locally in git folder and don’t persist changes outside this scope (cache). That is a good point for leaving the dependencies locally.

Haven’t you ever felt that your development environment is dirty? Or an eternal doubt whether you should install a library for testing and mess up everything?

VM’s are a feasible solution for working with different setups, but I am sure that not everyone can afford all the licenses and costs that it comes with.

Keeping Local (per project) Dependencies

I’ve always had a preference for keeping the dependencies locally, and taking advantage of the relative linking (I will demonstrate the power of it later).

The usage of local dependencies also allows you to change between projects with different versions of dependencies, and start working “right away” after you clone and pull the dependencies.

That is what is lacking this in LV, a project package manager. Yes,if you ask, NI Package Manager is a system package manager, something else.

From other languages we can get inspired in package managers such as pip, npm, go modules, yarn, conan, maven, and many others, but each of them with their scope and most of the times they are language oriented.

Can we have used some of them for LabVIEW?

Maybe, but then we would end up with too many modifications, and it would be unfeasible in a long run.

Furthermore, the need for more reproducible and faster jobs at automation tools are coming to LabVIEW. Containers (docker) is probably the next Must. Keeping dependencies local makes it easy for job executors to cache dependencies and make better management.

You could disagree, but that is actually what most of the newest package managers of the top used programming languages do.

Git Based Projects

We’ve been keeping our projects in git, we’ve been using versions, we’ve been tagging them. Why do we need to create packages of source code? Why don’t we use tags for it?

See, right now I am not writing about Compiled Code (PPLs) yet, only source code. Thinking of that, I’ve seek a solution across the internet and did some experiments.

SDM, a first approach

It all started at the beginning of the pandemic, I spent some time in it, but I failed miserably in trying to create something similar to a dependency manager. This is a theme that when you start digging, you will never stop, it is another academic degree.

So, I decided to take an easy path, and later, I would think about going further. Some glimpses can be found here on this post and on this post.

The SDM stands for Source Code Dependency Manager, which doesn’t manage inter-dependencies, but it downloads your dependencies, the workflow was the following one:

While the SDM solution was crappy, it proved myself during almost 2 years that the workflow is possible and it could be used for LabVIEW Projects. That was at least my great learning of trying to create some custom system.

Although, I haven’t given up on the idea (more about that later), I was wiser to spend more time doing some internet research, after all, I am not that smart to have the ultimate idea before everyone.

PERU, a language agnostic solution

PERU stands for … Better read their website for the description. Well, it was designed to be a generic (language agnostic) package manager for git based projects. And finding this project, let me think that I could reuse this project for LabVIEW project.

For my surprise, it actually overwhelmed my expectations, PERU does more than just downloading your dependencies. It actually handles other types of VCS such as SVN, Mercurial (Hg), and ordinary file types. All you need is Python and a per project configuration file.

pip install peru

Source Code Dependency Example

This is probably one of the common use cases for every developer.

You write something, then you feel like this code could be reused somewhere else. The code is beautiful, well tested, but you don’t know how to reuse it on other projects. Maybe you don’t know if you want to reuse at all (read this post).

The first approach is, just copy it, and include in the project. But then, you have an untracked copy of the original project. If you need some update on the original one, how would you keep all other copies up to date? With one or two projects, you could do it manually. But with more, than you are in trouble.

If I were in a Python project, than it would be just updating my reusable library by (one of the methods):

twine upload dist/*

Of course. After following all the versioning steps. Then your project, you could just do:

pip install your-reusable-package —upgrade

Let’s put out the theory and see one example using peru and LabVIEW.

First I created my first piece of potential reusable code. It is called Lib-A, it does one thing. It sums two elements.

https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/lib-a

Afterwards I create my second one. It is called Lib-B, it does one thing. It multiplies two elements.

https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/lib-b

Now I have a main project, and I want to use both projects in my new “main” project. Using peru, I am pulling them into my main project into the lib folder in this example.

https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/main-project

# First section defines the destination of the "packages"
imports:
  lib-a: lib/
  lib-b: lib/

# Second section defines "packages" url, tags or commits hashs and optional filters to remove partially the files. 
git module lib-a:
  url: "https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/lib-a"
  rev: v1.0.1
  export: src/

git module lib-b:
  url: "https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/lib-b"
  rev: v1.0.1
  export: src/

Run the peru command.

peru sync

Great. Now I have already them as dependencies in my project, without doing anything else (Don’t forget to “git ignore” .peru and the folder you chose for storing the dependencies.)

In this case I used tags, but you can go deterministic and use git hashs.

• Updates to dependencies:

In some cases, you expect to update your dependencies. You could just edit your file with new dependency, and run again a “peru sync” command.

In this example I did a breaking change on one of the dependencies (changed connector panes). Using per project dependencies make it easy to compare, because I can open projects side-by-side (of course, they need to be in different folders).

• Open Source Projects

Let’s suppose that I want to include some open source projects, in this case I chose two projects from two LabVIEW Champions (Jörg and Sam) that I know that follows good Software Engineering Practices.

https://gitlab.com/sas_public/write-anything-tdms

https://gitlab.com/hampel-soft/open-source/hse-logger

I also choose those projects, because I know they do not depend on many other packages, which it would require extra-effort mapping all of the deps. So, I add few more lines on my peru.yaml.

imports:
  hse-logger: lib/
  tdms-write-anything: lib/tdms-write-anything

git module hse-logger:
  url: "https://gitlab.com/hampel-soft/open-source/hse-logger"
  rev: v2.0.6
  export: Source/
  drop: Source/hse-logger.lvproj

git module tdms-write-anything:
  url: "https://gitlab.com/sas_public/write-anything-tdms"
  rev: v1.0.3
  submodules: false
  export: TDMS Headers/

Voilá. I have two open source projects on my project (maybe I need to mass compile or something due to LV Versions), but usually it is okay.

Binary Code Dependency Example

Getting back to our first example. I later decided that one of my libs is pretty stable, and I want to create a compiled library of it (lvlibp). So, I did it and publish as a Gitlab Release.

-git module lib-a:
-  url: "https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/lib-a"
-  rev: v2.0.0
-  export: src/
+curl module lib-a:
+  url: "https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt/lib-a/-/package_files/29218825/download"
+  filename: "Library A.lvlibp"
+  sha1: ece7fd3a0db118de76a9562a088d70b6829b02fc

I did in more than one step, because I used the “Replace With” function in the project, where I needed both libraries in the lib. But you got the point.

Making it easy and some caveats

For making a bit easier when transitioning between branches with different versions of the dependencies, it is important always to close LabVIEW.

Not only in this case, but this a necessary practice because you can’t keep changing files in disk that were already loaded on memory. Possibly a problem for you.

For handling peru better, I create a git hook (post-checkout) script:

#!/bin/bash

## Author: Felipe Pinheiro Silva <felipefoz@gmail.com>

TRIES=0

DEPS_FOLDER="lib/"


set_permission(){
  if [[ -d $DEPS_FOLDER ]]; then
    cd lib/
    if [ $1 = "true" ]; then
      attrib +r '//s'
      echo "r"
    else
      attrib -r '//s'
      echo "rw"
    fi
    cd ..
  fi
}

run_peru(){
  set_permission false
  peru sync
  set_permission true
}

# Verify Peru and try to Install
check_peru_install() {
  if type "$1" >/dev/null 2>&1; then # if given a valid command
    "$@"                             # run that command with original arguments
  else
    if [ $TRIES == 0 ]; then
        TRIES=1
        echo "Peru is not installed. Installing it..."
        check_peru_install pip install peru
        run_peru
    else
        echo "Pip or Python is not installed, please install before using peru generic package manager!"
    fi
  fi
}

## Argument 3 is the flag for defining if is a file checkout or branch checkout
## We are interested only on branches (value == 1)
if  [ $3 == 1 ]  &&  test -f "peru.yaml" ; then
   check_peru_install run_peru
fi

Additionally, it turns the folder that I choose as a “Read-Only” folder. You could add other routines there such as mass compiling, for example, that would take some extra time, but that’s your choice.

Also, LabVIEW versions are not verified, so you need to make sure you are working with a compatible version.

Flat dependency hierarchy is not supported either, so you could leverage a peru.yaml file inside your dependencies and download them as well (in fact they are, but only inside the project, and not exposed so clearly, there is a workaround, but things start getting too complicated).

Peru’s curl module (file download) still does not support authentication, so until it does, we cannot use private repositories download.

SDM v2?

Thinking of that, maybe a v2 of SDM could be possible, more like a wrapper for LabVIEW around peru, including LV version check and everything else, including project integration.

Just maybe, but that’s a completely new project.

Some future ideas

This post should trigger plenty of ideas on people reading it, but one idea that I had, is that a wrapper around peru could leverage Open Source projects through a common index (like PyPi), that could be hosted in GCentral.org for example.

A nice integration with LabVIEW Project would be interesting, but that would require some extra effort, due to unloading and loading of VIs into memory.

I think that’s all for today. I might have forgot something, but then I will just edit this post later.

The source code for this post is available at:

https://gitlab.com/felipe_public/felipe-kb-blog/git-dep-mgmt

Please, leave your comments if you found this interesting or doubts if you haven’t understood something.