+1
Completed

Partial quick unit database update

chrwoizi 3 years ago updated by Tom - Henry Coursow 3 years ago 13 2 duplicates

Whenever a C# script is changed, we need to run "Update Unit Options" from the Tools menu to update the definitions for flow machine nodes that reference custom scripts. This process is taking about 90 seconds in my project. It iterates through all types and rebuilds the whole database. 

Most of the time, I have only changed one script to make it compatible to Bolt or to fix a bug. I don't need Bolt to reimport all types from my code and from other assets that my project uses.

It is possible to roughly detect script changes (via C# file dates) and to generate and update the unit options only for those types. This would be way faster than the full rebuild (probably less than 5 seconds total) and, although not perfect, would suffice in most use cases. At least for workflows like mine, where I am the only developer and am integrating Bolt for specific tasks into an existing C# centered project.

I have implemented this feature in my project as a new Tools menu item "Update Unit Options partially" using types from the Bolt DLLs. A description of the code can be found in this thread:

Custom script method not shown until update unit options

Bolt Version:
Unity Version:
Platform(s):
Scripting Backend:
.NET Version (API Compatibility Level):
GOOD, I'M SATISFIED
Satisfaction mark by chrwoizi 3 years ago

Duplicates 2

Pending Review

Hi chrwoizi, welcome to the community! :)

This is normal and an "automatic fast update" is unfortunately impossible. Unity can send a notification when the project gets recompiled, but it does not exactly specify which classes, methods, fields or properties were changed/added/removed. For this reason, the only way to update to unit database is to analyze the whole codebase at once.

This is, as you noticed, a long operation, which is why it's not done automatically on every script change. There are plans to make it faster in the future, currently scheduled at version 1.1.5 in the roadmap, but it still won't be near-instantaneous.

Maybe, after a unity recompile event, you could iterate through all c# project files and compare their modified date to the database date. That should give you a quick list of all changed scripts within some milliseconds. Then you could refresh only the class definitions that match the filename, asuming the developer uses the common convention that a file name should match the name of the class that it contains.

That's an idea, but it wouldn't work for:

  • DLLs
  • Types that get removed
  • Generic types
  • Types that do not match the file name
  • Multiple types in a single file

Because of this, it cannot replace the complete unit generation. If you want, you can submit an Idea thread for "partial quick unit database update". Considering the database does not associate types to scripts though, it would require reworking the database storage format.

Is there a reason in your workflow why you need to rapidly change the codebase and change the Bolt graphs that use it at the same time? 

The reason for this request is that I already have a fairly large Unity project where all logic is implemented in C# and I want to implement the AI in a visual programming language. I often need to change properties and methods (e.g. make them public or change parameters) to make them compatible with Bolt. The wait time adds up and is very inconvenient. 

I guess, the problem here is that I am a one-man-show C# programmer who also wants to do certain things in Bolt. Other developers may fall into other categories which are better supported by Bolt, e.g.

- The "designer" who just wants to do a little visual coding and sticks to DOTween or similar plugins without ever writing a C# script.

- The large company that distributes tasks between many employees, one being the coder who commits his code to git at the end of the day and one being the AI programmer who updates the unit database in the morning and works exclusively with Bolt.

I found a way to partially update the classes that match the name of changed files. The regular update takes 90 seconds. My partial update takes 20 seconds and it could be way faster ( <5 seconds) if I had a way to filter types inside UnitBase.GetStaticUnits. 

I don't want to paste the full editor script here because it is based on the source code of UnitBase.CacheStaticUnit and I'm probably not allowed to distribute that unless Lazlo gives permission. But here is a description of what my script does:

Directory.GetFiles("Assets", "*.cs", SearchOption.AllDirectories)

Compare their System.IO.File.GetLastWriteTimeUtc to System.IO.File.GetLastWriteTimeUtc(BoltFlow.Paths.unitOptions).

This lists all changed C# files and takes only milliseconds.

Then I remove all options from UnitBase.staticOptions where the type is IMemberUnitOptionLiteralOption, or ExposeOption and the target type is in the list of changed C# file names.

Call UnitBase.GetStaticUnits() and keep only units of type MemberUnit, Literal, and Expose where the target type is in the list of changed C# file names.

Build a list of unit.Option() and add them to UnitBase.staticOptions.

Serialize those options with .Serialize()

Open the database with SQLiteConnection.

For each changed C# file name: "DELETE FROM UnitOptionRow WHERE unitType IN ('Bolt.Literal', 'Bolt.Expose', 'Bolt.InvokeMember', 'Bolt.SetMember', 'Bolt.GetMember') AND (tag1 LIKE '%." + className + "' OR tag1 = '" + className + "')"

Insert the previously serialized options.

File.SetLastWriteTimeUtc(BoltFlow.Paths.unitOptions, DateTime.UtcNow)


The caveats listed by Lazlo (DLLs etc.) still apply, but for my use case this is negligible.


I would like to see the partial update integrated in a future version. If that's not possible, can we at least get a way to filter by type inside UnitBase.GetStaticUnits() to make my partial update routine faster?

-1

On the roadmap, Lazlo does intend for there to be better connection references and changes. Not sure how far down, but this should significantly improve in the future.

Under Review

I definitely agree it's a hassle, and here are some things I hope to do to alleviate that:

  • Improve connection error recovery so that connections can stay even if the source or destination unit fails to load
  • Allow unit replacement, meaning you'll be able to switch back to the new unit after a change, and connections would get reinstated (Done in v.1.2)
  • Add support for an attribute such [FormerlySerializedAs] on types and members
  • Look into refactoring tools (really big endeavor, only at the research stage at the moment)

The idea to automate this process even further (e.g. 1 button click, choose new type) is not bad, I'll think about it!

+2
Completed (Unreleased)

The next version will include an experimental incremental unit database update!

It is very, very fast. Basically, Bolt checks MonoScript GUIDs and diffs them with their last write time, as chrwoizi suggested. But because it doesn't necessary look for types or file names, but really GUIDs, it supports DLLs, generic types, etc. 

This works for:

  • Adding new types or members
  • Removing types or members
  • Deleting types or members

The caveat, however, is that your types must be properly organized according to the Unity standard, meaning:

  • The type name has to match the file name
  • There cannot be multiple types in the same script file

Otherwise, Bolt will exclude the types during its incremental updates (but still recognize it on a full build, which may be confusing).

The menu has been reworked. A "full rebuild" is now called Build Unit Options. An incremental, fast rebuild is now called Update Unit Options.


This new rebuild is so fast that it can reasonably be done automatically in a background thread every time a codebase change is detected. To enable this, just toggle a checkbox in your preferences:


As indicated, all of this is experimental, so please let me know if you encounter any issues!

Nice! Thank you there is an Settings option to do it automaticly?
Was quite annoyed by doing it manually all the time...
Thanks :)