Managing Dependencies in SCC – Part 2 – The Solution

So, in the first post of this issue, I wrote about what problems and issues I had and what I was seeking to solve them.

As I told before a co-worker told me that Gitlab has some extended features and then he introduced me to the Gitlab API.

Then I started my quest of developing a LV Code to interact with this API.

Git API

It turns out that most of the majors hosting git repositories/platforms have a REST API, in that includes Gitlab, Github, Azure Devops, Bitbucket, AWS Code Commit.

If you have no idea what is a REST API, I suggest you by start reading this link.

Just summing up, it is a model (architectural style) that most of the web uses for synchronous requests protocols, exchanging information between a client and server.

LabVIEW itself has an API for dealing with such HTTP Requests. JKI also developed over this API a toolkit for simplifying the requests and responses, which are very tricky to deal with.

So, to have an idea of the scalability of this solution, any public Git repository of any language could be used as a dependency. I mean, if it is private and you have access then it also can be used.

What is a tag?

The mandatory recommendation is to have a tag versioning system compatible, in this case, Semantic Versioning.

A tag is a point referenced to a commit in your source code control system. So it is not your commit per se, it is a fancy name for it, instead of using the long commit hash, you use i.e. “v1.0.0”.

Most of the developers already use tagging and something very close to semantic versioning, so changing it is not a challenge, and different from the code itself, you can change tags and apply retroactively, although most of the time it is not a good idea, it can be done.

Dependencies and Project Organization

After studying for a while the API, doing some tests with LabVIEW and command-line interface (using curl), I came up with a rudimentary version (quick and dirty of the code). This was, of course, the test version.

I realized that a problem would have to be solved first: the cross-linking in LabVIEW. For example: if your library depends on the other library, and that library depends on a third library. But your main library depends also on the third library. What a mess, isn’t it? This diagram will help you understand better.

The main code downloads a dep1 (One) and dep2 (Two). The dep1 also downloads dep2. So now we have two dep2 in two folders, one within the main library and another within the dep1 folder.

The point is: to solve this, I came up with a solution that turned out to be a good choice for standardizing projects hierarchy in our company (future post on it).

My solution was to put all the files that I want in a folder called “libraries”, so the dependencies would be always in parallel folders relative to each other, and that solved my issues with libraries depending on each other, code duplication and cross-linking.

Implementation

After playing around with Gitlab API, I came up with the second solution on that first one (can I call it quicker and dirtier?). It has the following features:

• Read all the projects in the Gitlab;
• Retrieve all tags
• Allows user to select tags;
• Generates a JSON file with these tags;
• Check for JSON file in the selected tags;
• Download the source code into libraries folder;
• Put dependencies libraries into read only mode.
• Insert dependencies in .gitignore to not be uploaded to SCC;

I thought that a video would be self-explanatory, so I recorded a small video executing the process.

Version 2

When I decided to write post I had also put the second version in a planning step. Quicker and Dirtier became a headache, many bugs and not handling correctly errors. But for this, I thought that at first, it would be good to raise the Requirements, “Pros” and “Cons” of the viable solution that would require minimum effort, based on the current solution.

Pros

• Use an already settled infrastructure (Git Hosting Plataforms);
• Easy to understand;
• Needs only tags and a json file in the main project;
• There is a command line for quick downloading;
• Any project can be used as dependencies (python, c++ libraries, etc…);
• NXG Compatible (not the app itself, but the downloading);
• It checks for updates;
• Written in G;

Cons

• Not fully developed and tested yet;
• To use it, folder structure refactoring in the projects is needed;
• Written in G, so, run-time engine needed (not that bad, after all it will be used by LV developers);
• Does not handle LV Version issues;
• Do not handle completely dependency versioning. Although it downloads only one of each, it does not guarantee that the versions will be inter-compatible.

Looking at these points, it really got me motivated to continue this project, but I thought the requirements could go further. If people embrace this idea, it could be possible to turn this project open-source. Here are some required features (and risks) to allow everyone else to use it.

Desired Features and Risks

• Allow any Git platform API – Maybe doing some class inheritance/interface to change between different providers. The APIs are different, although required functions exist in most of them;
• Extend command line, not to only download, but do some extra configuration;
• Extend JSON configuration files, to include other information such as licence, authors, contributors, copyrights, LabVIEW Version;
• Include automatically libraries in the lvproj (VI Server). A plus, not really necessary.
• Allow reading/downloading assets in Releases Page for people who use Packed Libraries (lvlibp).
• It would rather have a local cache to avoid too much downloading and increase speed in setting up big projects;
• Show projects Readme and Changelog.
• It would depend on someone else’s repository (you could prevent yourself by removing those files from .gitignore and uploading to your SCC)

Verdict

Well, this solution, for now, is really useful for me. I can move between my projects without worrying about dependencies. I really see the possibility of expanding this project, imagining that you would just need any repository link to list tags and include in your project.

But, it is just a thought of a community project, if it has enough feedback, we may keep talking and advancing on this agenda.

Would you use it?

What features are missing?

Let me know what you think in the comments below.